Development / Edge / IoT / Security / Contributed

Effective Strategies for Open Source Supply Chain Management

7 Jul 2021 9:02am, by

Jeremy Katz
Jeremy Katz is co-founder and head of engineering at Tidelift, where he’s scaling the infrastructure powering the company’s software platform. Before Tidelift, he was a software engineer at Google, Stackdriver (acquired by Google), and HubSpot. Jeremy started in software engineering at Red Hat and was a long-time contributor to and board member of the Fedora Project. He earned his master’s degree in systems and design management at MIT and a BS in computer science at North Carolina State University.

If the recent SolarWinds breach and the ransomware attack by DarkSide on Colonial Pipeline have taught us anything, it’s that software supply chain attacks are a real and serious concern. It should be at the top of an IT leader’s list of priorities.

In the case of Colonial Pipeline, the largest fuel supplier between Texas and New Jersey, it wasn’t the oil company’s core operational technology (OT) that was compromised but its IT systems. We don’t know the exact method used, but we can be confident it likely exploited an old vulnerability in software that didn’t receive the same attention as critical OT systems. We do know the ransomware literally targeted the company’s wallet by compromising the automated ticketing system for billing customers. By exploiting the IT system, DarkSide forced the oil company to shut down its pipelines to contain the threat, and suddenly half the fuel supply for the East Coast of the United States was in danger.

We know more regarding the cause of the SolarWinds episode, and it demonstrates the danger and ingenuity of bad actors targeting the software supply chain. Many standard security measures were negated in this instance by the fact that the software build process itself was subverted.

In the SolarWinds attack, a malware payload called Sunspot was inserted into source code before the code was compiled into a program and signed for distribution and use. A high-profile attack like this usually spawns a stream of articles reaffirming the need to keep on top of your software updates, but that’s not the answer if the latest signed update is the update that’s compromised. It was only by monitoring the software’s behavior that the breach was eventually uncovered — but remaining undetected for months is not a great success story for any security product.

Some pundits have pointed to the “closed” proprietary nature of Orion, the IT monitoring software, that the bad actors used as the foot in the door to start collecting data from SolarWinds’ customers. It’s true that Orion’s code wasn’t readily available and transparent for anyone to review, which certainly didn’t help investigations.

But that doesn’t mean open source software is magically immune to such an attack. Today, anyone building applications with open source software must have a strategy in place for effectively managing their organization’s open source software supply chain.

Responsible Use of Open Source Requires Responsible Management Practices

Open source has had its high-profile vulnerabilities too. The Equifax hack, which exposed the personal data of 143 million accounts, was only four years ago. That exploit was made possible because Equifax was using an unpatched, out-of-date version of a popular open source web framework, Apache Struts 2. The cause was the bad practice of ignoring critical updates, which Equifax later acknowledged.

The following year, open source experienced the Node.js event-stream exploit, which was closer to SolarWinds in the sense that the attack vector was the event-stream source code itself. The cause was a bad actor who had simply asked for and been given commit privileges by the original creator-maintainer (who had ceased actively maintaining the project himself). In the case of event-stream — which was a popular package with over two million downloads — the attacker decided to add a malicious version of a library to the source code. The target was a specific type of cryptocurrency wallet, which the exploit would attempt to steal and send to the attacker’s server. This attack was clever and only detected because it used a deprecated crypto module that created an unexpected warning elsewhere.

The event-stream exploit highlighted a primary risk inherent to organizations that are not proactively managing the open source software that flows into their applications. Historically, there is no consistent standard for how open source projects are maintained — which means there are no traditional service level agreement (SLA)-type assurances like those an organization would typically receive with proprietary software.

Meanwhile, because of its many compelling benefits, open source has only grown in popularity and pervasiveness. Developers love open source for good reason: it enables them to be more productive, reduce costs, and accelerate development and time to deployment. A recent Tidelift survey found 92% of modern applications contain open source components.

Balancing the Need to Move Fast and Stay Safe

Because open source has become the de-facto choice for application development, every organization needs a strategy to ensure it can move fast and stay safe when relying on open source in its applications.

Most organizations currently manage open source in one of two ways:

  • A distributed approach that puts the priority on moving fast. Developers on each team can pull in the open source components they need from third-party sources as they need them. This approach allows them to build and deploy quickly, but the tradeoff is a lack of consistent management practices, increasing potential security or maintenance headaches.
  • A centralized approach that prioritizes staying safe and reducing risk. Developers don’t choose packages individually; instead, they have a centralized process requiring time-intensive approvals before using new components. This approach often employs scanning tools that block deployments until developers address flagged issues. While this strategy helps prevent security vulnerabilities from slipping through to production, it slows development down, adds friction, and frustrates developers.

Thankfully, organizations don’t need to choose one of these two extreme approaches. The middle ground is an approach to managing open source that automates policy governance and streamlines processes for bringing in and approving open source components.

This middle-ground approach is centered around organizations building a centralized catalog of known good, pre-vetted open source components that meet a company’s current and future needs.

Implementing a Strategy for Open Source Supply Chain Management in Phases

We see organizations typically grow into an effective strategy for managing open source over time.

Often, the first step is to appoint an open source strategy lead. This should be an established and informed member of your organization, such as an enterprise architect or director of engineering, with the credibility and clout to get the organization aligned around its approach to managing open source.

Next, the open source strategy lead will convene a small group to draft an open source vision and charter that reflects and complements the enterprise’s business goals. This document will define how open source plays into the company’s strategy and future goals. It will define policies about usage, risk tolerance, and organizational contributions to open source projects, among other things.

Once some of the core open source policies are in place, the next step is to get clear visibility into the open source already in use at the organization today. You’ll want to establish an inventory of open source components in use (also commonly referred to as a software bill of materials, or SBOM) and ensure you have defined standards around security, maintenance, and licensing for those components.

Developers need an established “paved path,” so curation of approved components is vital. Teams need clarity on, for example, whether they can use a package or not, what the procedure is for reporting a bug in open source code the company uses, how they can request a new package, and whether and how they can contribute to open source projects during work hours. Without this, a decentralized approach can fester, with no consistency between teams, and leaving individual team members unsure about what they are “allowed” to use and not use.

You will also want to ensure the organization has a basic understanding of open source licenses and a point of view about which licenses are acceptable for use. Knowing the differences between the popular licenses, such as MIT, Apache 2, and GNU GPL v2 and v3, is key to selecting the right components for use in your application development projects and avoiding licensing-related risk.

Without adopting a solid open source management strategy, an organization can expose itself to significant legal, security, and technological risks.

When best-in-class open source management practices are used, open source components are proactively maintained and secured, and clear standards and policies are set — helping open source software becomes a competitive advantage rather than an unmanaged risk.

Feature image via Pixabay.

A newsletter digest of the week’s most important stories & analyses.