Zero CVEs? Don’t Compromise Quality for Easy Compliance
Let me tell you a secret. When my CI/CD kicks off and the vulnerability scanner outputs a “zero” on the vulnerability check step, my face instinctively draws a smile. But it takes only a few seconds before I return to reality. Anybody who has been in software engineering long enough knows that achieving zero security flaws, also known as common vulnerabilities and exposures (CVEs), is a Sisyphean task.
I’ve noticed a lot of buzz around attempts to achieve zero CVEs of late. That’s understandable. But, at the same time, its value is debatable. In my view, the potential effects of this zero-CVE chatter (read: obsession) are detrimental. But by following some sustainable practices, you can minimize CVEs and the harm they can cause, while maintaining software quality.
Vulnerabilities Are Not Going Anywhere
Let’s face it: vulnerabilities exist everywhere. They might even be present on the commit you just pushed into the stable branch of that one project you’re most proud of; you just don’t know about it yet.
Maybe the stars are aligned in your favor right now. Your shiny new application doesn’t happen to have any known vulnerable component, hasn’t been targeted by attackers and hasn’t received unexpected input or actions that cause it to fail.
While I sincerely hope nothing of the sort ever happens to your application, it’s always a very real possibility. The truth is, until a magical AI tool is able to create perfect code for us, vulnerabilities will continue to arise in the code we write and the components we use to build software applications.
Compliance-Driven Security: The Dangerous New Norm
Meanwhile, governments are taking very serious and appropriate steps to start enforcing better security across software supply chains. In the United States, the recent Biden-Harris National Cybersecurity Update and its predecessor, the Executive Order on Improving the Nation’s Cybersecurity, are proof that we are headed in the right direction. These initiatives are asking the right questions of open source software (OSS) vendors in a bid to strengthen their security practices and improve security standards across businesses and industries.
So, institutions are definitely noticing, which is great. However, this has also given rise to an awkward and risky new approach to security: the idea of “compliance-driven security.” When reducing the number of CVEs (even unfixable ones) becomes a top goal for software engineering teams, they’re given a dangerous incentive to take shortcuts to satisfy auditors, whether they be people or software (scanners).
It’s very telling that the most popular talk at the latest CloudNativeCon was about how multiple vulnerability scanners could easily be fooled into thinking that a container image didn’t have any vulnerabilities while, in reality, it had more than 100.
Don’t get me wrong here. Reducing the number of CVEs is a critical and healthy software engineering practice. But it’s important for that reduction to come organically, keeping a focus on quality. If compliance adherence becomes an obsession, not-so-good practices will inevitably be adopted to achieve it. If the objective is to reduce vulnerability noise at all costs, you will sooner or later find yourself in a situation like driving with your headphones on — there will be noise, you just won’t be able to hear it. Everything might seem bright and shiny until that zero-day happens. And at that moment, perhaps your scanner isn’t telling you the truth or isn’t telling you anything at all or maybe it’s telling you just what you want to hear.
A Quality-Driven Approach to Minimizing CVEs
The fact that vulnerabilities will always exist doesn’t mean we need to resign ourselves to a fate of unsecured software. There are many things we can do to have as few vulnerabilities as possible. As the team behind the open source Bitnami Application Catalog and its enterprise version, VMware Application Catalog, my colleagues and I have learned a thing or two about minimizing the security threats associated with using open source software. Open source Bitnami-packaged applications have accounted for billions of downloads by developers on multiple hubs and marketplaces while we also privately deliver built-to-spec, production-ready OSS images to customers using the enterprise version.
These are some best practices we’ve learned that can help you take a quality-driven approach to minimizing CVEs in a healthy, sustainable way.
Use Minimal-Yet-Useful Base Images
While distroless images remain an eternal promise, very often, shipping the application runtime and code is not enough. Yes, the image size will be very small, but it is equally important to find that balance between a reasonably small threat surface, usability and production readiness. The utilities that come along with the major Linux distributions like Ubuntu, Red Hat Universal Base Image or Debian are a good reason to consider using them as the base OS, and they all have slimmed-down versions that some even brand as distroless. An even more compelling case is their long history of support, seasoned security processes and strong communities. VMware’s own Photon OS is another option you can consider using if you are looking for a well-maintained, minimal and reliable base image.
Continuously Update OSS Images
It goes without saying that using the latest versions of OSS is critical for minimizing the threat landscape and ensuring a strong security posture. Find ways to ensure that you are aware of all upstream releases of the OSS project you use and get them into your system as soon as possible. As you scale up OSS usage, it might not be feasible for you to ensure that all your OSS applications are continuously updated. That’s when you might want to turn to vendors that can provide you with continuously maintained OSS images.
However, there has been recent debate around whether enterprises should wait for the upstream software vendor to fix a vulnerability and release a new version or patch as opposed to finding a third-party vendor that can get the vulnerabilities fixed for them.
It’s true that upstream fixes don’t always happen in the smoothest of manners. It might take a few days for an upstream software vendor to release a security patch for their application. But in all honesty, when and how to fix a vulnerability is their decision to make, because who is a better judge than them to make a call on how critical a patch is?
Directly modifying the upstream source code can also result in some undesirable consequences, like loss of warranties or inadvertently introducing new, unknown security risks or functionality issues. So, perhaps the most important step in sustainable OSS adoption is controlling what you can by ensuring that you use the latest releases and patches as soon as they are released upstream.
Choose the Right OSS Vendor
Minimizing CVEs and their impacts on your security posture requires a strong commitment from the upstream software vendor as well. For each app development use case, developers often face a wealth of options in the OSS ecosystem. So, selecting the right software requires due diligence and a careful analysis of the risks posed by each potential choice.
Our team maintains an internal checklist to review all potential additions to Bitnami Application Catalog and VMware Application Catalog, which ensures consistent quality, security and conformance. I recommend having a similar checklist, based on your enterprise policies, to review an OSS project’s governance, licensing, security and operational practices and history, which can provide insights into the health of the project and clues to any adoption risks.
During this process, if you happen to notice a pattern from an upstream OSS project ignoring security vulnerabilities, you should reconsider the trust you have placed in that application provider and re-evaluate your options. There is never a dearth of options in the OSS ecosystem.
Software Bills of Materials Delivered in Standard Formats
One practice that’s gaining prominence and that helps enormously with CVEs is the software bill of materials (SBoM). SBOMs help you get complete visibility into a list of an OSS application’s dependencies, such as third-party services, software packages, open source software stacks and codebases. SBOMs provide an exhaustive inventory of software components, allowing you to seamlessly track application changes and identify and remediate vulnerabilities.
SBOMs are more relevant now than ever, especially with the prominent role they play in the aforementioned U.S. government mandates. Mature ISO standards like Software Package Data Exchange (SPDX) and CycloneDX are fantastic battle-proof formats that enterprises can use to ingest software provider data into their own tools like Graph for Understanding Artifact Composition (GUAC) to gain additional knowledge about your software ecosystem.
Actionable Visibility into Vulnerabilities with VEX
If you aim for a perfect zero on your CVE reports and feel paranoid about the presence of unfixed CVEs but want to do better than simply instructing scanners to ignore those CVEs, you might want to evaluate some fresh initiatives, such as the Vulnerability Exploitability eXchange (VEX). VEX is a new standard that provides machine-readable vulnerability assessments from software vendors or distributors. By getting these vulnerability assessments in an international OASIS standard like CSAF or other formats like CycloneDX, OpenVEX or SPDX (SPDX 3.0 will include VEX support), you will gain actionable visibility and context into how your workloads are affected by vulnerabilities, helping you reduce the real threat footprint.
Mind the Basics
Finally, no matter how few CVEs your OSS container images or open virtual appliances (OVAs) might have, it’s irrelevant if your OSS components do not meet the basic security requirements of modern software. Making sure that your OSS Helm charts are compatible with the two latest major releases of all Kubernetes cloud providers, verifying that all your OSS containers are running as non-root, ensuring that your applications will rebalance after autoscaling and that they can be upgraded easily are just a few of the many checks that you or your software provider should be doing.
Always Look to Quality
Yes, looking to build software with zero CVEs is good. But what’s better is ensuring that you don’t lose out on quality in the process. While improving the security posture of your product or platform, remember to focus on quality rather than on easy compliance. Good engineering practices will always be the best way to achieve secure, sustainable consistent adoption of OSS across multicloud environments.
To learn more about quality-driven security and how it can help mitigate risks from upstream vulnerabilities, register for this live webinar on Nov. 28, or watch the replay of this talk by VMware Field CISO David Zendzian and Forrester Principal Analyst Sandy Carielli about why you need to stop treating security as an outcome.