The threat of duplicative work can keep IT managers awake at night. The imagined murmuring of discontented stockholders or the accusing finger of the CTO can bring cold sweats to the most-hardened engineering directors. Faced with the threat of wasted effort, many managers have beaten the drum of code reuse, advocating an inner source culture that encourages developers to share their code with other teams in their organization.
It’s thus easy to imagine the frustration of an engineering director who, after years of such advocacy, finds development velocity is actually trending down instead of up. Their teams struggle because they’re unable to modify dependent libraries. Services break when shared components become incompatible. Teams don’t even understand their own code and have started to fear change — despite how change is the lifeblood of a successful IT organization. Before long, the director knows an upstart disruptor will be along to eat their lunch.
If this situation sounds familiar, you might be facing a similar problem: a broken strategy for sharing code.
As a senior software engineer at New Relic and former consultant, I’ve seen organizations wrestle with the pain that inevitably comes from rapid change and growth, and I’ve seen teams successfully adapt and thrive. Although no two organizations are exactly the same, the following five diagnoses and suggested remedies can help engineering managers and enterprise architects regain any velocity they may have lost to high-friction code sharing strategy.
1. The Tragedy of the Commons
If your organization has many repositories of unowned and inconsistently maintained common code, something is wrong. Most likely these “commons” or “utility” repositories don’t meet the quality standards you’ve set for the rest of your engineering organization. Everybody is accountable for them; therefore, no one is.
- Dozens or even hundreds of libraries live in your repos and artifact repositories, many of which haven’t been touched in years;
- Many of these shared assets are used in services you ship, but the people who wrote them, or know how they work, have moved on from your company long ago;
- Some libraries have no owner. Some have several owners. Either way, they are a point of friction between teams. Because there has been little attention to what does and doesn’t belong in the libraries, they’ve have become collections of code that do many different things, often only tangentially related to each other;
- Libraries are rarely deprecated or deleted, and the teams that consume them are either unaware or uninterested in whether they can or should upgrade the libraries they use;
- Dedicated individuals work long hours outside of their team’s core discipline to keep libraries functioning — and are burning themselves out.
- Swing the pendulum from reuse towards autonomy. Make teams fork any repositories that aren’t owned and then eliminate them. This will spark some duplication, but it will also break the logjam until you can push those common capabilities into a mature platform;
- Create standards for libraries. For example, require that any shared library has a clear owner who must publish service-level objectives (SLOs) for the library;
- Require all teams that publish shared libraries to have a product manager accountable for the libraries’ management;
- Break up common libraries and distribute them to the teams that most logically align with the purpose of the libraries. Teams then either maintain the library or provide a path to the same functionality in some other form;
- Encourage teams to share knowledge instead of libraries. Code samples and demos can often provide the same result as a shared library without creating inter-team dependencies.
A collection of shared libraries may have made sense when your organization was smaller and everybody really did own everything. But as your company grows, these unmanaged shared dependencies become problematic. You should be able to stop the commons from growing, but it will take some time to eliminate the last of the shared resources.
2. Transitive Dependency Hell
Transitive dependencies often come along for the ride when a developer uses an internal or external framework or library. For example, using Spring Web will necessarily involve Spring Core, Jackson, Netty and a host of other third-party components. Similarly, Node.js developers who decide to use Express will have to pull down 30 libraries authored by a number of different developers. These types of dependencies are hard to avoid, but they can become a point of friction if allowed to get out of control.
- Teams are highly resistant or unable to upgrade internal libraries and frameworks because these libraries depend on a latticework of transitive libraries that create irreconcilable incompatibilities and require substantial QA efforts just to test minor changes. As a result, the time to mitigate serious bugs and security vulnerabilities grows unacceptably long. Sometimes the libraries don’t get patches at all, forcing intervention from security teams;
- If one critical shared library has a dependency change, teams are forced to spend time performing sweeping library upgrades, again consuming significant QA resources;
- Teams simply don’t know at any given time what code they are running because dependencies shift underneath them. When a library breaks, the team has little recourse but to dig in and figure out what first-order dependency brought in the transitive dependency, so they can fix or pin the correct version.
- Require publishers of shared libraries to minimize the dependencies their libraries require. How closely you need to follow this rule depends on the nature of your organization. At one extreme, if you need to add a library to your library, the scope is already too big. A softer line might be more appropriate for many organizations;
- When transitive dependencies are unavoidable, require teams to move them to a private application-specific namespace. Depending on your platform, this could be relatively easy or very difficult. If you use npm, for example, this is already done for you. If you’re building a JVM-based application, you’ll have to take some extra steps, such as creating shaded jars.
Modern software applications are a web of third-party commercial and open source components, with a thin layer of custom domain-specific glue your developers wrote holding it all together. It’s unlikely you’ll change this pattern any time soon, but you must eliminate the tensions between your teams; deep transitive dependencies increase the tension dramatically.
3. Coupling, Cohesion and Fragmented Concerns
Coupling and cohesion may seem like ivory tower concerns, but if left unchecked, they can slow engineering velocity. Coupling refers to the interconnection of two items, including classes, components or even teams. Cohesion is an indication of the logical relationship between the parts of a module.
Excess coupling makes it difficult for connected items to change independently. Low cohesion makes it harder to understand the module and creates coupling between unrelated items within it. Further complexity arises when individual concerns are fragmented across files, classes or services. The result is that change is difficult, slow and high risk.
- Deploys across teams need to be carefully coordinated because a single concern is fragmented across team boundaries or multiple capabilities are entangled within a single deploy;
- Changing one dependency involves changing a large number of its dependents to prevent them from breaking;
- Configuring a single use of a dependency involves making a change in many places.
- Advocate for small-and-compact, single-purpose units in any shared code. Remove any libraries that are just grab bags of function; look for files called “util” or “common,” for example. Preach the Do One Thing and Do It Well gospel in communities of practice. Be wary of framework proliferation, and when you do add new frameworks, make sure they address truly foundational concerns and aren’t a reflexive mechanism for preventing duplication;
- Push more intelligence into services instead of relying on client libraries. If your teams spend a lot of time negotiating feature additions to client libraries, fork them instead and let each client team move in their own direction. If you want to use client libraries as APIs, rigorously manage them as products, making sure they’re sufficiently flexible and configurable;
- Tightly base APIs on the business domain, and don’t let them leak abstractions. Ask yourself if your APIs address the right concerns or if your service teams needlessly work across domain boundaries;
- Be wary of false analogs. If you see two teams working on similar projects, be sure they are truly analogous before pushing them to share components.
Organizations with problems of high coupling and low cohesion often struggle with bigger problems than just a suboptimal shared-asset strategy. However, reducing cross-team dependencies and increasing team autonomy can give your organization more flexibility to course correct as business needs shift.
4. Version-Proliferation Hell
We’ve all seen it before: a team makes a tool or framework available, thinking of its contribution as just one asset. But months later, team members are busy maintaining who-knows-how-many versions of the same basic code. Welcome to the hell that is version proliferation.
- Teams that provided a shared resource are drowning trying to maintain many versions of the same basic code;
- Bugs and security vulnerabilities are fixed quickly on the most-recent version, but unpatched versions run rampant through your organization because teams consuming the library can’t or won’t upgrade.
- Shift broadly shared logic out of libraries and into services. Service owners can then decide the upgrade cadence;
- Whether your teams are building libraries or services, practice careful version management to minimize the number of versions teams have to maintain. For example, make API signatures backward- and forward-compatible. Avoid required fields, accept unknown fields and maintain version information inside payloads to reduce conflict across API versions.
Releasing shared code should carry with it the weight of responsibility. Upfront thought and planning can help teams prevent that responsibility from exploding and inflicting pain on themselves or the rest of the organization.
5. Distributed Accountability
Here’s an exercise: gather your team in a circle and throw a glass vase up in the air. One of two things will happen: no one will lunge for it, and it will shatter on the floor, or more than one person will lunge for it and they crash into each other. (Actually, it’s probably better to think of this as a mind exercise since throwing a vase in the air is probably an HR liability.) The lesson? Ambiguous accountability inevitably leads to teams dropping important work or duplicating efforts.
- Teams create a lot of code they don’t run, so they can’t easily upgrade.
- Teams run a lot of code they don’t own, so they can’t easily evolve.
- Work with teams to clearly define the boundaries of their domains. Identify areas of substantial overlap, and try to find ways to reduce it by defining producer/consumer relationships and eliminating shared code or libraries that don’t belong. This can be difficult and time-consuming, particularly the first time through, but it will become less disruptive and more effective if you make it part of your software development process;
- Encourage teams to demonstrate their knowledge rather than package it in a shared library. Accept you’ll have some duplication in order to create clear ownership boundaries.
In large, complex organizations, blurry boundaries around accountability and ownership lead to confusion and errors. Distributed accountability of shared libraries is not only a symptom of that confusion, it also deepens the pathology. You can’t totally eliminate dependencies between teams, but it helps to clearly define the nature of those relationships. Rigorously working toward clear ownership models will lead to fewer cases where an artifact is owned by no one or everyone.
Even organizations of moderate size and complexity need to be conscious of the costs of duplicate logic and effort; but a hasty solution could be worse than the problem. Seemingly inexpensive approaches, such as sharing code without investment in product management practices can result in increased coupling, ambiguous ownership and ultimately, reduced team autonomy and productivity. These issues are insidious because they tend to get missed on the ledgers until they’ve become a crisis. And by then, it may be too late to correct your course.
New Relic is a sponsor of The New Stack.
Feature image via Pixabay.