This story is the latest in a series that we will post that examine the role of the security in the open source software supply chain.
One of the biggest attractions for automating application operations is that automation tools will always be able to respond faster than a human intervening manually. This is true for tasks like scaling and failing over — and it’s also the case with responding to security vulnerabilities.
“At the end of the day, the risk that many organizations face in their infrastructure and in their applications is ‘what if there’s a security vulnerability exploited before I have time to react,’” said Aner Mazur, chief product officer at open source security company Snyk. “They’re talking about patching.”
Patches are not always the best long-term solution to security issues, but they can fill in the holes while organizations figure out what the best fix is — which might be using a different operating system, changing tools or updating workflows. But when a common vulnerability and exposure (CVE) is found, organizations need to react as quickly as possible. That means patching, and the more automated the patching, the faster it can be done and the shorter the amount of time the organization is exposed.
At first glance, automated patch management would seem like the no-brainer way to solve this problem. As soon as a CVE is discovered, the system would automatically do the update or patch and fix the CVE in minutes while the security team continues sleeping through the night.
The reality is slightly messier. Security risks have to be balanced out with operational risks. Sometimes downtime risks outweigh the security risks involved with taking extra time for manual intervention when it comes to patch management. “The industry has yet to find a silver bullet solution for this problem,” said Om Moolchandani, co-founder and chief technical officer at DevOps security company Accurics.
So how does automated patch management work? When is it a good idea, and when is it not? I chatted with a couple security pros to find out.
How Does It Work?
The idea behind automated patch management is simple. “I have some CVE that shows up in my software somewhere, and rather than having to go in and manually review it and figure out stuff like what version I need to upgrade to in order to get rid of the CVE, I just have automation in place that does it for me,” explained Hillary Benson, chief product officer at container security company StackRox. Simple, right?
But where exactly would the automation tool sit? How exactly should it work? “My general feeling is a sort of one size fits all one tool to rule them all kind of approach is unlikely to be successful just because there are so many variations in how people deploy and run software,” Benson said.
Snyk provides automated patch management, and Mazur said that user feedback clearly indicated that the CI/CD pipeline is the best place for automated patch management. In their process, if a patch is available for a CVE Snyk will automatically trigger a pull request and a git code change. A developer would be alerted to the patch and would approve the patch before the build kicks off.
Benson wasn’t so sure that automated patch management can be limited to one part of the workflow. “It’s like the integration of all of your systems,” she said. “I don’t think an automated patch management system can necessarily sit in any one particular part of your environment and actually cover the full scope that it needs to.”
In the Real World…
You might notice something about Snyk’s automated patch management… it is not fully automated, because it includes a step where a developer manually approves the build. “One of the feedback we’ve gotten is that customers want the automated patching, but they want to approve the patch,” Mazur said. In other words, truly automated patch management makes people nervous.
There is full agreement that automated patch management isn’t always a good idea. “Especially in scenarios where you have mission criticality above security, automated patch management is not going to serve much of a purpose,” Moolchandani said.
The reality is that security is about risk management, and while automated patch management can improve an organization’s security posture it can also increase operational risk.
The reality is that security is about risk management, and while automated patch management can improve an organization’s security posture it can also increase operational risk. Using automated patch management wouldn’t be wise if the application is so critical that a minute of downtime would cause dramatic revenue loss.
The operational risks involved with automated patch management also depend on the type of infrastructure you’re using. “If you deploy and manage your infrastructure manually, automating patch management is not reasonable,” Benson said. “So the more automated your entire processes of deploying and managing running software, the more automated your patch management can be.”
If you have a brittle, legacy application, automated patch management is not a good idea. “The further along in the journey to cloud native infrastructure you are, the easier it is to find ways to automate patch management in a way that’s actually feasible and repeatable and less risky,” Benson said.
Are You More Secure?
“One of the biggest misconceptions is that automatic patch management can make you secure,” Moolchandani said. “Security is always a journey. It’s not a destination. Automatic patch management increases your level and makes you better in the security game, but it’s not the be-all of security.”
Yet automatic patch management is clearly essential to maintaining a strong security posture, even if the process is only 75% automated, it still allows organizations to respond much faster to vulnerabilities. “The number of vulnerabilities that are released is so high that if you were to manually patch all of them, you just couldn’t match the speed,” Moolchandani said. “It’s very important that you patch often and you patch quickly to stay one step ahead of the attackers.”
According to Maner, one way the industry might move more towards completely automatic patches is by applying machine learning to determine if a patch is “safe.” If, for example, the patch failed one CI build-out of a thousand, that patch would be safe enough to deploy without manual approval. Users would be able to configure what percentage of failed builds is safe enough and then completely automate the patching.
Accurics and Snyk are sponsors of The New Stack.
At this time, The New Stack does not allow comments directly on this website. We invite all readers who wish to discuss a story to visit us on Twitter or Facebook. We also welcome your news tips and feedback via email: firstname.lastname@example.org.