DevOps is all the rage these days. Everyone is talking about it, and companies are jumping on the practice to automate their development and deployment processes. However, DevOps is complicated, and there is no one right approach ideal for every team. How you should measure success will also depend on your situation, and there are so many potential mistakes to be made when deciding what metrics to use.
Measurements and metrics are related ideas, but I make a distinction between the two. I use measurements when I talk about individual items that have numbers or values associated with them and metrics when I talk about measuring results or determining whether or not something is successful. For example, one measurement might be the frequency of releases, which could be used as a performance metric to determine whether you are getting new features to your customers more quickly.
I like to think about metrics and measurement using a few questions:
- Why are are metrics important?
- How should I decide which metrics are important for my situation?
- What should I measure?
While I’m looking at these as three separate questions, they are interrelated, and you’ll see that in practice, there is quite a bit of overlap when thinking about metrics and measurement as you embrace DevOps.
Why are Metrics Important?
Measurement in DevOps is not something new, of course. In 2010, very early in the days of DevOps, Damon Edwards and John Willis started talking about the idea of Culture, Automation, Measurement, and Sharing (CAMS) as fundamental values of the DevOps movement. Lately, I seem to see more people talking about culture, automation and sharing than measurement. But, measurement is important, and figuring out what to measure and how to measure it can be tough for many teams, especially ones who are just beginning to embrace DevOps.
Gene Kim writes about DevOps principles and “creating a culture that fosters two things: continual experimentation, taking risks and learning from failure; and understanding that repetition and practice is the prerequisite to mastery.” To learn and improve on your way to mastery, you need to track the metrics that will help you determine whether you are making progress.
“If you can’t measure, you can’t improve. A successful Devops implementation will measure everything it can as often as it can … performance metrics, process metrics, and even people metrics.” — John Willis, July 16, 2010.
The only way to prove whether what you are doing is making a difference is by measuring and tracking your progress over time. Ideally, this should start with tracking some baseline metrics from before you implemented a change to a process or activity and continued well after the change to understand the impact over time.
Which Metrics to Use?
The key is to understand why the change you are about to make is important for your business before you decide which metrics to use to track it. You can spend way too much time measuring things that don’t really matter and calling your newly adopted DevOps strategy a resounding success while your company crashes and burns around you because you aren’t focused on the right metrics.
From my perspective, this is the most important of the three questions because this is where you decide which metrics matter the most for your organization. It’s critical to think about what your organization as a whole needs to accomplish to serve your customers and how your team fits into that bigger picture. The metrics that you choose to track should be directly related to the business goals of your organization.
This is also where the culture and measurement values of DevOps go hand in hand. If you want to make sure that your metrics are tracking the right things, developers and operations need to work closely together with the rest of the business to understand the bigger picture. You’ll notice that this goes beyond just getting developers and operations teams working together as if that wasn’t difficult enough in some organizations, but it also means having discussions with the teams running other parts of the business. By creating a culture of open discussion about the needs of the business, you’ll not only ensure that you have the right metrics, but it’s also likely that you will better meet their needs in other ways, too.
What Should Be Measured?
Now that I’ve talked about the importance of selecting the right metrics for your business, it’s time to think about how measurement and metrics are different, and why it’s important to distinguish between the two.
For metrics, pick a few that make the most sense for your business, and track those closely. These metrics are probably the ones that you’ll show to management when you talk about how embracing DevOps has made a difference, not just for your team, but for the business as a whole.
However, for your team, you’ll probably want more data and more details. While I recommend keeping metrics focused on a few things, I have the opposite stance for measurement. Measure as much as you can because you never know what is going to be important in the future. You’ll want baseline measurements for things that you didn’t know were going to change. You’ll also want to look for trends in areas that might need to be improved later. This is especially true for performance measurements since changes in one area are likely to have positive or negative performance impacts in other areas. If you aren’t looking at a wide variety of measurements, you might miss some trends that will turn out to be important later.
This all boils down to a few related ideas summarized into these three key points.
- Metrics are important. This is how you can tell whether you are successful, and hopefully, demonstrate this success to your management team.
- Measure more than you think you need now. You never know what you’ll need later or what trends you’ll uncover until you start measuring beyond what you know is important today.
- But, only track metrics for what’s important to your business. You’ll need to have developers and operations working closely with other business teams to determine which metrics are the right ones for your organization.
Pick your metrics wisely if you want to avoid being the one surrounded by explosions and fire proclaiming, “This is fine.”