Measuring Engineering Velocity: Deploy Frequency as a ‘Vital Sign’ of DevOps Health
If you’re in the middle of adopting DevOps at your organization, you’ve no doubt run into some common roadblocks, like organizational friction and cultural inertia. Getting around these challenges requires measurement: without it, you won’t know how to track or define progress, and you can’t be sure that your digital transformation efforts are moving in the right direction.
To help define and improve DevOps measurement, our new CircleCI report looks at three metrics most likely to affect engineering velocity: mainline branch stability, deploy time and deploy frequency. We based our findings on a sample of GitHub and Bitbucket organizations built on CircleCI’s cloud platform in mid-2017. In this article, we’ll focus on our results for deploy frequency; read more about mainline branch stability in our previous article and deploy time in the next article.
Deploy Frequency: A Key Indicator of High Performance
Deploy frequency indicates a company’s actual speed, and is also a function of stability and deploy time: high mainline branch stability and low deploy time both encourage higher deploy frequency. We look at deploy frequency as a “vital sign” of an organization’s DevOps health. With each “pulse,”or deployment, an organization is delivering value, discovering customer needs and fixing problems. Frequency is also directly affected by cost: When the cost of a deploy goes down, engineers are encouraged to deploy more often.
“We see that deployment frequency is a key indicator of high-performing organizations,” says Dr. Nicole Forsgren, author of Accelerate and CEO/chief scientist at DevOps Research and Assessment LLC. “High performers can deploy on demand, while their low-performing peers can only deploy their code once per month or longer. This difference in deployment speed makes all the difference in delivering value, delighting your customers and keeping up with compliance and regulatory changes.”
Findings: In our study, deploy frequency was measured as the median number of “default-branch” builds run on our cloud platform with a valid deploy step, per week per organization. We found that 75 percent of all organizations deploy their most active project less than 13 times a week. Top performers (those in the 95th percentile) deploy their mainline branch 32 times per week; that’s over five times the median and nearly 24 times the bottom 5th percentile.
For organizations in the top 10th percentile of Alexa Internet Ranked organizations, we saw an increase at the leading edge and an even greater spread, with the 95th percentile deploying 42 times per week — 40 times the bottom 5th percentile, but still five times the median (eight deploys per week). A slight difference in range is also reflected, with 75 percent of organizations deploying less than 16 times a week.
Top Alexa Internet Ranked organizations appear to move most quickly. There is a slightly higher distribution between high-scoring organizations and the rest, with the top percentile of Alexa Internet Ranked organizations pushing code more at the high end. This is a reflection of the importance of velocity: to be the best in any category, organizations must maximize their deploy rates.
Best Practices for Maintaining Deploy Frequency
Small, short-lived pull requests: Deploy frequency is a composite metric, dependent on both mainline branch stability and commit-to-deploy time (CDT). One well-known but powerful tactic is to limit the size and lifespan of pull requests.
“Generally, we have a culture against long-lived branches, to the extent that around 80 percent of them get merged within 24 hours,” explains Jeremy Stone, head of engineering at Code.org. Brad Buchanan, lead software engineer at Code.org, adds, “We’ll see pull requests as small as one or two lines of code, and a bulk of them fall under 200 lines.”
While this might seem impossible, it shows Code.org’s commitment to splitting work into small, shippable units. By doing this, Code.org created a culture of incremental development, allowing engineers to get code out of review and into users’ hands more quickly.
Feature flags: Another way to shorten pull requests (PRs) and increase deploy frequency is to use feature flags. “Large PRs are large because of dependencies,” says Tim Wong, principal technical account manager for LaunchDarkly. “By shipping dependent pieces independently of each other, you reduce the amount of code in PR rot.”
Breaking work into digestible chunks is at the core of continuous software development. Feature flags empower developers to ship work they’re doing as they complete it. Developers are happy because they’re finishing tasks, and users are happy because their product still works.
Deploy frequency is one of the best predictors of an organization’s velocity and growth (along with mainline branch stability and deploy time). By tracking this important metric, you’re on the road to developing best practices to improve deploy frequency — and spreading DevOps principles throughout your organization.