Developer-First Security: How Comcast Cut Vulnerabilities
“Improving daily work is even more important than doing daily work,” wrote IT guru and CTO Gene Kim in his book, The Phoenix Project. That maxim can improve an organization’s security posture when developers apply it, contended Larry Maccherone during the virtual conference DevOps Connect: DevSecOps Con sponsored by TechStrong.
“Think about that for a second: It’s less important to do your daily work than it is to do things that will improve your ability to do your daily work in the future. That learning effect is so much more powerful,” Maccherone said during his presentation, “The 3 Ways of DevOps as the Keys to Developer-First Security.”
W. Edwards Deming called it “continuous improvement” and used it as a model to revolutionize manufacturing. The underlying idea is that by involving those who do the work in improving the work, organizations can reduce effort down the road by eliminating errors and problems. In application development, vulnerabilities are often left to pile up until the end of the development cycle and are often given to a developer who didn’t code them in the first place. This makes them harder to fix than they would have been earlier in the coding cycle. That’s why Maccherone recommended a more integrated, continuous approach to development and security.
“You can’t say every feedback from a tool is just a checkbox I need to check and as soon as I fix that vulnerability, I’m done,” Maccherone added. “No, you have to think of every feedback from every tool, every incident production, as a learning opportunity and you have to focus on preventing the next one of those.”
Addressing Security During Development
Maccherone conducted an informal audience survey, asking where the bottlenecks are in security. He gave a binary — was it detecting vulnerabilities in the application code or resolving them? About 80-90% said the latter, he noted.
“It’s resolving them. Why then do we constantly look for a better tool that will find more things?” he asked. The norm is to roll out a vulnerabilities tool, scan all the apps and then find out “how bad it is,” he said. It’s overwhelming and it leaves vulnerabilities as a separate project, to be identified and fixed at a later date.
Instead, what he introduced at Comcast is the idea that developers should identify and fix vulnerabilities sooner, while they’re still in the “flow” of development. That might mean only identifying a subset of critical vulnerabilities at first go, he added. It’s a practice he’s cultivating into a framework for web security.
“Pick one team and one application from that one team, pick one tool … and turn the policy dial down so low that the team is willing to commit to resolve all those vulnerabilities in one or two sprints,” he advised.
Then turn up the heat a bit.
“Once you do that, you put in place this branch protection status check in the pull request that first just says, ‘Hey, you’ve got these, these criticals in your application,’ and then as soon as they get clean the first time, you switch it from a notify feedback — where it’ll just basically be an orange checkmark, and it won’t block the merge button — to a required status check,” he said. “You can always override it for good business reasons.”
At the same time, install a second status check — a feedback mechanism that isn’t blocking but that tells the team how many “high severity vulnerabilities” (or whatever the next severity set of vulnerabilities is) the application has. It might not even be high, he added, if that’s more than they can finish in the next two or three sprints. The point is to ensure the policy dial on software composite analysis (SCA) is two or three sprints ahead of the development team.
“In general, I think it’s easier while you’re working on SCA feedback to get the criticals and highs clean before you move on, …and then you can think about mediums or other tool categories after that is the progression,” he said. “If you do even that first one of those, I think you’re going to address a huge percentage of your risk, maybe 40% of your overall risk just doing the first one of those. So you get complete value out of it.”
By comparison, there’s a net negative value of running the scan across all applications if the development team doesn’t have time to resolve them, he said. Plus, when developers try to fix code after launch, it’s more cumbersome: The task may fall to a different developer, who will have to spend more time researching the code and project than the original developer would have if the vulnerability were identified during the application development process, he explained.
Measuring Progress for DevSecOps Success
Metrics played an important role in the program, not just to justify it, but to provide feedback that improves the work.
“At Comcast, if we couldn’t show that we had been more effective for less costs — with the data and the feedback we were getting — we wouldn’t have gotten the go ahead to expand to the next round of development teams,” he said. “It is really important that you capture metrics just to gain executive sponsorship.”
It also helped developers and engineers identify what was working and to what extent it was worth the effort. By measuring the impact on vulnerabilities, the development team could see what was working. What they found was that developers weren’t just fixing vulnerabilities — they were actually reducing vulnerabilities during the coding, he said. Ultimately, that’s more important than just talking about best practices.
“I contend that’s not nearly as important as coming up with a way to get people to adopt any set of good practices,” he said. “And if you use this feedback mechanism, they’ll eventually be the perfect set, the best set.”
