What Good Security Looks Like in a Cloudy World
This is the first in a three-part series on continuous security.
In the fast-paced world of today’s modern software delivery, velocity, is for the most part, the chief concern of engineering organizations. This focus on velocity above all — in the form of time to development and, eventually, delivery — has been the catalyst for many changes in engineering processes over the last two decades. This has made significant changes across the board in engineering fundamentals, from processes to architecture, operations and management, and even tooling and systems.
These include the transition from waterfall to agile and lean methods, to breaking up the monolith into microservices, migrating to the cloud, CI/CD and automation, and much more. This was also a major contributor to many of the “as code” engineering practices elite organizations use today — Infrastructure as Code, Configuration as Code, Pipelines as Code — and the next phase in this transition that has yet to be realized is Security as Code.
Transitioning engineering practices to “as code” practices ultimately makes automation possible — transforming formerly manual processes and frameworks to automated processes with little human error. This has proved to be the key to unleashing the velocity and delivery businesses today require. This is also a primary reason for security being left behind in this transition, as finding the ideal ways to automate and simplify security into agile software delivery is an engineering domain that is still being shaped and defined.
The way security has been employed to date has made this only partially possible — with DevSecOps and shift left starting this transition. Yet, with all of the advancement in tooling and research that has been accomplished in the security domain over the same two decades, it is now becoming possible to begin unlocking the full potential of continuous security, which can and should become a native part of high-velocity engineering.
While security is often perceived as, figuratively speaking, putting the spokes in the wheels of engineering, in this post we will dive into why continuous security will ultimately be the enabler of high-velocity engineering for forward-thinking engineering teams.
The Security Breakdown and How to Fix It
Security starts to break down when it bogs down engineering teams, as it introduces many new vulnerabilities and fixes, which all receive equal priority. One thing we quickly understood at Jit was that for security to undergo this shift and empower engineers to own security, there needed to be three critical pieces to security management:
- Differentiation
- Prioritization
- Remediation
Together, these will enable engineering organizations to achieve a good baseline for what good security looks like in the long term and to continuously maintain this baseline. This is what we call “continuous security,” the next phase in automation — CI, CD and now CS.
Once this is understood and integrated into developer workflows and processes, organizations will be able to benchmark security, maintain this posture and even improve it without losing context or velocity.
Differentiation
So what do we mean when we talk about these three pillars? Let’s start with differentiation. When it comes to security issues and fixes, it is extremely important to be able to differentiate between new and old findings because this will also eventually affect the next two pillars: prioritization and remediation.
One of the things DevSecOps tools have made possible is a real-time understanding of what’s happening in our code, with processes aligned with developer workflows, such as fixes at commonly accepted gates, like pull requests, and even earlier with precommit hooks or in-IDE alerts. A similar approach to the way we prevent issues from being merged into our code base through common CI gating can be applied to runtime-related tools during the CD phase. In this way, you can prevent runtime-related issues from reaching production, as well.
So if we are able to discover security flaws while we’re still coding or in predeployment to production systems, these can be handled now and within the developer or operational context and need never go into the backlog. This is a very important distinction between our categories of security issues.
Once we handle issues immediately, and while developers are still in context and working on their code, we can then move on to older issues, those related to code already running in production.
Prioritization
By prioritizing existing vulnerabilities through intelligent noise filtering, you can help developers remediate with actual solutions when possible. The world of security engineering can take pointers from similar paradigms, like the Inbox Zero methodology or Eisenhower’s Urgent/Important paradigm.
Where this gets complicated is with backlog management. We want an easy and reliable way to surface the important issues in our backlog. This starts by having a way to first aggregate, log and filter the many outputs from the diversity of security tools in your chain based a prioritization mechanism that would take into account a number of different parameters:
- Whether there is a fix.
- If there is a fix, what is the level of effort required to make the fix (man-hours, cost and impact on production, which is particularly important if test coverage is only partial).
- Is it possible to aggregate and perform batch operations on similar findings?
- Severity of the issue.
After we figure out the priority, severity and ability to fix the issues in our backlog without any breakage, we can move on to the last and final piece that enabled dev-owned security, the actual remediation.
Remediation
Elite engineering teams, quantified by DORA metrics, are those that have low MTTR (mean time to restore) metrics and change failure rates. This means that they have a fix-first mindset. They won’t get bogged down with what went wrong in a crisis; they’ll get moving quickly on fixing it, and then only after do the required post-mortem and root cause analysis.
Security can learn a lot from this approach in systems engineering. Security also needs to get to a place of fix-first security. Tools today introduce a lot of noise and long lists of vulnerabilities found in your systems (code/infrastructure/runtime) with few of them providing any guidance for the best method to remediate or mitigate the issues. This is a passive approach to security, and it needs a rethink. We need to start taking an active approach to security for it to truly provide value and deliver on the promise of good baseline security hygiene.
Those remediation methods need to align with developer workflows such as pull requests and pointing to the exact line of code or configuration that needs to be fixed. By providing this kind of remediation insight, you will enable your engineers to start by first “stopping the security bleed” without the overhead of having to research and understand each issue individually, which often requires security expertise.
The prioritization phase provides the ability to derive a list of tasks you need to fix today, this week or this month. As we’ve already stopped the immediate bleed, the assumption is that we are doing our best not to introduce any new issues into the backlog when possible. Another mindset change security needs to undergo is to not create noise around unfixable findings. For an engineer, if there is an issue that can’t currently be remediated or fixed, it just creates unnecessary noise. Therefore these types of findings, while important for security engineering to know about and track, shouldn’t be escalated to developers until a fix is available.
Continuous Improvement
Once we have built a good process and workflow to resolve new and known findings with our existing toolset, we can add layers and controls on top of it and have a process in place to continuously improve our security. So like a minimum viable product (MVP), where our first iteration is just a starting point and we constantly build on top of it, the MVS (minimal viable security) approach works in much the same way. Build the most basic security controls to provide the baseline security you need for your products, and once you have a good process and workflow around detection, prioritization and remediation, you can continuously improve your security posture.
Fostering a Continuous Security Mindset
What the continuous security model introduces is a new way to think about how security is handled within engineering organizations. This is a good framework for all engineers to take greater ownership of security by embedding it into their processes and providing sufficient context and tools for remediation. Once this process is well established, it sets a foundation to continuously build upon and iterate your security posture, much like agile product management — in the minimal viable security mindset.
By creating a culture of dealing with new issues first and ensuring your issue backlog isn’t growing, you can then create a prioritization and remediation plan for existing issues so you aren’t in a constant rat race where you aren’t making any security progress. Once you’ve created a baseline security posture by stopping the bleed, you can then set out to continuously improve it.
In our next post, we’ll talk about continuous security from the CTO’s perspective, and the pillars it consists of that take it from theory to practice and ensure it is a truly viable next phase in the evolution of continuous everything, CI/CD, and now CS.
