A Guide to Navigating Open Source Support Requirements in the US Government
If software is eating the world, then open source is eating software. Linux is the dominant operating system, and progressively more enterprise applications are being built with high percentages of open source code. In response, the U.S. government has prioritized the adoption of open source software (OSS) in federal agencies. Teams in the national security and defense agencies have long been using OSS for cybersecurity and rapid response software-oriented problem-solving. That said, many agency IT managers are still uncertain about how to use OSS while complying with government mandates. In particular, there is confusion about support requirements.
Similar to U.S. government policies on software supply chain security like Executive Order 14028, the Department of Defense (DoD) offers specific guidance on OSS support requirements. To summarize, DoD development and IT teams implementing OSS must have a “plan for support.” It’s common for the DoD to set trends in the federal government that are picked up by other departments, so all U.S. government IT employees should take note of this policy.
While the DoD’s policy leaves a lot of flexibility, allowing agencies to choose from many options can, in practice, become a little murky. Even for agencies outside the DoD, it’s worth taking a deeper look at this policy to consider how to address the supportability of OSS.
What Is a DoD Plan for Support?
In the complex landscape of software management within government agencies like the DoD, a plan for support is an essential document that outlines how your OSS will be maintained, updated and secured throughout its life cycle. Along with the recent mandate for a Software Bill of Materials (SBOM) and other transparency mechanisms, a plan for support serves as the roadmap for sustaining the software’s integrity and functionality.
Broadly, there are three viable options for a DoD plan for support:
- OSS Vendor — Dedicated commercial support by an OSS vendor
- Known Contractor — Bundled commercial support by a contractor as part of a project
- DIY — Internal support by the DoD agency team
Notably, community support is not offered as an option. Although it might be economical, community-only support is not viable for critical operations. Community support often lacks the timeliness, expertise and accountability required by government agencies for national security and other sensitive functions. Moreover, community support doesn’t provide service-level agreements (SLAs), which are crucial in a setting where downtime or security lapses can have significant consequences.
Let’s break down each of the three viable options (plus a bonus) and weigh their pros and cons.
Option 1: OSS Vendor
Vendor support is similar to proprietary support, except the vendor signs a support contract with the DoD agency to maintain and support an OSS distribution. A prominent example of this model is Red Hat Enterprise Linux being supported by Red Hat.
Pros of Vendor Support
- Strong Expertise — In this scenario, the vendor usually maintains the OSS product, meaning support staff can be deeply knowledgeable about the software and may add significant value in terms of performance and additional security guidance.
- More Timely Notifications — When the vendor is also the maintainer, customers often get notified of critical updates and patches faster than if support is provided by a third party. Further, the vendor is often involved in the security process, providing unofficial access to security vulnerability information before it is made public. This ensures you can act quickly to maintain security and performance while aggressively mitigating risks.
- Value-Added Services — Deep expertise of a vendor also enables higher quality of value-added services such as training, architectural design guidance and configuration audits.
Cons of Vendor Support
- Higher Cost — Vendor support might require a substantial financial commitment over many years. The vendor needs to be able to plan and assign resources (which likely includes U.S. nationals assigned to the agency), so this can result in a subtle form of vendor lock-in.
- Lack of Dedicated Support Engineers — Multiple layers of customer service may mean there’s no single, dedicated contact for your needs. This can lead to lack of institutional knowledge, variable response quality and slower response times.
Option 2: Known Contractor
Contractors in government technology services are happy to offer bundled support for OSS as part of ongoing master services agreements (MSAs) and other contracts. This is already a multibillion-dollar business with many prime contractors having teams that deploy OSS tooling and products as part of their ongoing system integration work.
Pros of Known Contractor Support
- Service Bundling — Support is often packaged with other services like system integration. This enables the support contract to be tightly integrated with the build and ongoing maintenance of the entire system, providing a holistic view and institutional memory.
- Lower Cost — Support is often an add-on that can be tacked onto an existing large contract for minimal additional costs due to the larger size of a contract. For smaller bundles, the cost benefits are less pronounced. In the best case, the support can be added with zero additional charge.
Cons of Known Contractor Support
- Delays on Patches and Security — A middleman in the update and patch process can lead to slower implementations. Also, the contractor may have to deal with a separate security team for patching and vulnerability issues. This can lead to scattered information and slower response times.
- Less Expertise — While contractors may be experts in OSS, it is hard to match the deep insights of the companies that are actually developing and maintaining the software. This is particularly important if the software includes many additional dependencies (libraries, packages, etc.), in which case deep understanding of the OSS ecosystem is critical.
- Contract Switching Risk — Government systems tend to outlive support contracts, so there is always the risk of high switching costs and loss of institutional knowledge when a contract turns over.
Option 3: DIY
Contrary to what many government IT managers might believe, you don’t have to negotiate or pay for an external support plan if you can design and staff an internal one that fulfills criteria. This can include a combination of internal support and maintainer support from actual project maintainers through a vehicle like Tidelift.
Pros of DIY Support
- Faster Response and Iteration — Internal teams can move quickly and be more responsive to organizational needs. There is less friction, and the agency should be able to iterate on support needs more quickly.
- Better Customization — Because internal teams are closer to the actual use cases of the agency and are fully embedded, they are more able to provide customization and additional configuration modifications to precisely meet the needs of the DoD entity.
Cons of DIY Support
- Requires Headcount — Whether it’s a contractor or a full-time employee, internal support options require live bodies to do the work. This includes setting up on-call coverage for critical systems.
- Less Access to Shared Resources — Unlike external support, internal teams may not be as easily able to benefit from shared resources. For example, you might be in trouble if a key member of your support team goes on vacation.
- Lack of Specialization — Agencies that go this route will probably have internal teams supporting multiple OSS components, making it harder for them to build deep expertise in each component.
Bonus Option: Hybrid Internal and OSS Vendor
While not covered in the DoD guidance, a fourth viable option is a hybrid plan for support that includes at least one or two internal resources partnering with a vendor to form a complete support team. This can make for the best of both worlds, providing the continuity and expertise of partnering with an OSS vendor while also enabling the speed and institutional knowledge of an internal full-time resource.
Pros of Hybrid Support
- Combined Benefits — This option combines deep maintainer expertise and early access to security upgrades with speed and institutional context of internal teams.
- Flexible with Strong Coverage — With two modes of support, this option is more flexible and can handle institutional change, turnover, vacation or round-the-clock coverage.
Cons of Hybrid Support
- Higher Costs — Unsurprisingly, this can be a costly choice because it includes a support contract and headcount.
Factors to Consider When Choosing a Plan for Support
When it comes to choosing a plan for support, the stakes are high — especially for DoD entities. Support of mission-critical systems is crucial to maintaining national security and the integrity of the DoD’s technology systems. A well-informed decision when choosing a support modality can significantly mitigate risks.
Consider these factors before making your selection:
- Dependency Level — How critical is the OSS to your agency or group operations?
- System Importance — Is the software running on mission-critical systems where failure or latency would have serious consequences?
- Usage Blast Radius — Is the OSS used across multiple teams or applications, creating a wider exposure for any problems or outages?
- Security Needs — What is your tolerance for time-to-patch of vulnerabilities and risks? Do you need internal or nearby experts who could potentially remediate a risk prior to a patch being issued?
Conclusion: Support Is Insurance
There’s a saying in the software world: OSS is free like a puppy. It may not cost you any money to adopt, but every puppy parent agrees that responsible dog owners should be prepared to budget time and money. Applying this concept to the realm of OSS and support, think of a support plan as an insurance policy. When your organization depends on OSS, support is something you must have.
That said, the quality and type of support depends on your risk tolerance. If you’re not too worried about the health of an OSS component, then internal or contractor support is probably adequate. But when that component represents a substantial single point of failure and the consequences of an outage are dire, then OSS vendor support (or a hybrid model) are a safer pick.
While more expensive doesn’t necessarily equal better support, keep in mind that investing in the right support option for your organization can have a direct correlation to time savings related to troubleshooting and outages. Choose wisely!