Why Securing Containers Is Like Going to the Dentist
Getting a routine dental checkup is something many people put off. Whether they’re a bit nervous about visiting the dentist, it’s an expense they don’t want to incur, or they’re just too swamped to fit it into their busy schedule, it’s a regular maintenance item that’s easy to postpone or deprioritize.
Securing containers is similar to routine checkups in that way. It contributes to good ecosystem health, but there are many reasons engineers put off the important maintenance of securing their containers. It’s important for IT management to understand this resistance, as well as the impact that not regularly hardening or patching containers has on application security.
Securing Containers Reveals a Skills Gap
Vulnerability scanners churn out a lot of data, which can be overwhelming for engineers, even those well-versed in security. The amount of noise compared to useful information reported is significant.
In many companies, engineers don’t have the security knowledge and training to grasp the gravity of critical vulnerabilities, let alone the know-how to locate and mitigate them. Mitigation can require engineers to make code changes or perhaps DevOps to make changes to the base container image they own, depending on the organization’s structure.
According to the 2022 Slim.AI Public Container Report, only 26% of developers are confident they have the knowledge to secure containers for production use. Developers and DevOps may recognize the need to patch certain packages, but packages are still rarely fixed. Security professionals must make significant time investments for triaging and routing vulnerabilities before they reach engineers, depending on the company’s risk appetite and organizational structure. This process can be quite painful, and as companies grow, triage and routing become harder and might take a significant portion of the time allocated in the service-level agreement (SLA).
Even when the triage process is efficient and can filter the most pertinent and actionable vulnerabilities, mitigation might not be a simple or quick task. For example, even if your scanner indicates that a patch is available, the patch might be available for an indirectly used vulnerable package, leaving the direct dependency unpatched. There go a few hours of an engineer’s time down the drain with no discernible product or security value gained. And it still may require a significant effort to replace the package you’re directly using. Frustrating!
If all the stars are aligned, and a patch is available for the package you’re using, the engineer may need to make code changes to align with the new package APIs to fix breaking changes introduced in the library. That assumes the test coverage is sufficient to prevent the code’s behavior from changing negatively. Oh, but wait — then you may find the same vulnerability in other places in the same container, sometimes still used indirectly by the code. More hours down the drain, less stability in the code, and no change in the scan vulnerability count. More frustration!
Indeed, dealing with a single vulnerability might be painful, and the frustration can quickly snowball over the vast landscape of containers, each harboring potentially thousands of vulnerabilities. Organizations may oversee hundreds of containers, culminating in an alarming surplus of wasted time, mounting frustration, and a concerning absence of tangible product value. To make matters worse, while it may take a “few hours” to do a minor package version update when you have consistent patching and package upkeep in place, that doesn’t come close to the time required to update a major version or replace a package nearing its end of life.
Drawing a parallel to the dental checkup analogy: If you don’t visit the dentist regularly, when you finally do, the visit will likely be much longer (and far more unpleasant).
Organizational Challenges to Container Updates
Setting aside the technical and knowledge gap challenges of doing package updates and addressing vulnerabilities, there are also serious obstacles from an organizational standpoint, especially in terms of prioritizing work.
Many engineers consider securing containers tedious work that doesn’t introduce product value. Even when security organizations push hard for resolution, engineering management may not prioritize or incentivize container security because security is rarely an engineering metric. “Good job, engineering, for not having a security breach and delivering less product value,” said no CEO, ever. Engineering rarely sees container security as an activity that impacts the bottom line. Therefore, it’s not usually considered a priority, unless security teams are exceptionally effective in surfacing it in measured doses.
That is true until a customer refuses to deploy a new version due to unresolved or newly discovered vulnerabilities. Company leadership then gets involved, and the issue is suddenly the highest priority as it directly impacts revenue.
Returning to the dental analogy: If you’ve neglected your routine checkups, you might need a root canal, and suddenly it’s an all-hands-on-deck situation. Not ideal.
Even engineering organizations (and CEOs) that put safeguards in place to avoid introducing new vulnerabilities during the development process (such as using pre-commit hooks) are not safe. New vulnerabilities are always being introduced and discovered, which puts a major burden on engineers to address them when they’re found, well after the code is developed and shipped. New vulnerabilities can appear in code that hasn’t been changed in a while, requiring significant effort and time that stalls all of the engineer’s other work.
Security Maintenance Best Practices
A better scenario is when a company is proactive about vulnerability remediation, communicates upfront to its customers about found vulnerabilities and explains how and when they’ll be addressed. A company that takes this approach will not only reduce surprises and fire drills and gain its customers’ trust, but it will also save a lot of work for the organizations that scan and rescan its containers as they’re released.
Taking the time for regular checkups prevents much more painful experiences down the road. Similarly, even though it may seem like a hassle to make small investments daily to keep the packages you use up to date, it’s far better than the alternative: If you neglect regular updates, the price you pay in time and breaking changes will be much higher.
The negative impacts of deprioritizing regular maintenance of container updates shouldn’t be underestimated. As the idiom goes, “An ounce of prevention is worth a pound of cure.” Here are some prevention and remediation strategies that can help your organization maintain a good bill of container health.
Shift Left for Detection and Prevention
- Scan everything all the time and as soon as possible. The best place is in the integrated development environment (IDE) while developers are working on features. This way, vulnerabilities can be identified as they’re introduced in the code or at minimum, as code is pushed to CI/CD.
- Scan continuously with multiple scanners to allow early detection. The key is to find vulnerabilities before your customers do (and it becomes a fire drill with upper management).
- Minimize unnecessary code by using smaller base images and multistep container builds and slimming — removing unused code from the container.
- Automate package updates, so you’re using the latest stable version whenever possible.
- Include a security evaluation as part of the design process as new packages are introduced.
- Provide ample security education for the engineers on the team so they understand the importance of maintaining the code and are empowered to make better decisions earlier.
Strategies for Remediation
- Improve routing by capturing ownership and container maintainers so you can assign remediation responsibilities to the team responsible for the type of vulnerabilities and package manager.
- Ensure that the security and engineering leaders collaborate well, have good contracts in place and use a proper triage process based on the system configuration. This includes noise reduction and annotations when applicable. Note that annotations are remediation.
- Make sure your CI/CD pipeline is modern and your software is thoroughly tested so patching is as quick and automated as possible without degrading software quality.
- Use well-maintained packages with reputable authors and a significant community. If you don’t have one already, add a vetting process for your supply chain dependencies that considers these factors.
Just like no one wants to be surprised with a dental emergency — especially one that could have been prevented with a regular checkup — engineering teams don’t want to be caught with vulnerabilities that could’ve been addressed with regular container updates. Engineering leaders need to prioritize the important task of educating and incentivizing developers to scan, automate, test and slim containers as a part of regular container maintenance. This is vital for the health of your ecosystem and, most critically, your customer relationships.
