TNS
VOXPOP
Favorite Social Media Timesink
When you take a break from work, where are you going?
Instagram/Facebook
0%
Discord/Slack
0%
LinkedIn
0%
Video clips on TikTok/YouTube
0%
X, Bluesky, Mastodon et al...
0%
Web surfing
0%
I do not get distracted by petty amusements
0%
DevOps / Operations / Tech Culture

Can Enterprise DevOps Ever Measure Up?

It's been 15 years since DevOps came on the scene. But for more traditional enterprises, why does their DevOps transformation seem never-ending?
Feb 6th, 2024 6:11am by
Featued image for: Can Enterprise DevOps Ever Measure Up?
Feature image by Image by Nedim from Pixabay.

We are more than 15 years into this thing called DevOps. Yet a lot of organizations seem stuck mid-transformation, with perhaps more friction than ever.

Part of the problem is that engineering is a science, so you can’t improve what you can’t measure. And measuring any kind of “transformation” is hard. It’s even more difficult since, despite its name, DevOps has focused much more on optimizing operations rather than the developer experience.

It’s not so simple to measure developer productivity. And, despite news outlets’ love of writing about DORA metrics, the SPACE framework and last year’s DevEx metrics, only a handful of companies seem to be actually using something similar — Microsoft and Google — the origins of most of this research — and then other big names like Netflix, Spotify, LinkedIn, Atlassian and GitHub.

Those at more traditional enterprises — that suddenly find themselves tech companies too — are left with a sense that they haven’t spotted the light at the end of the lengthy DevOps tunnel.

Of course, it’s not unusual for Big Tech to be the early adopters in the Hype Cycle, but DORA metrics turned 10 this year — no matter how interesting us pundits find them, it doesn’t seem like they are being widely adopted.

What is holding back these legacy organizations from becoming DevOps elite organizations? We speak with Christina Forney, Uplevel’s vice president of product, to learn just what’s holding them back.

Trapped in the Triangle of Pain

For the past 10 years, Forney has ebbed back and forth between product and engineering, always working on productivity tools for developers. The last five years were spent working closing with enterprise customers, where she witnessed this investment but little return on DevOps.

Most organizations, she said, are trapped in a “triangle of pain.” This is Uplevel’s name for the measurement of three key people health components:

  • People health — Are we working sustainably?
  • Commitments — Are we planning effectively and meeting our commitments?
  • Delivery — Are we efficiently delivering quality solutions?

“It’s really easy to do one or two of these three things. It’s very hard to do all three,” Forney said, “because you can meet your commitments with quality, but you’re going to burn out your team. Or your team’s really happy you’re meeting your commitments, but it’s taking a really, really long time, so quality is suffering.”

It’s fairly simple to balance two, but a stool needs three legs to stand.

Engineering leader painA triangle with three elements continuously flowing into and thus affecting each other: Commitments - Are we planning effectively and meeting our commitments? People health - Are we working sustainably? Delivery - Are we efficiently delivery quality solutions?

The Triangle of Pain, courtesy of Uplevel.

It’s important to note that it’s not a lack of understanding DevOps objectives that’s holding back enterprises. After all, there’s a multimillion-dollar consultancy industry making sure it’s explained. As Forney put it:

“We know we need to ship faster. We know we need to hear customer feedback. We know we want to deliver business value. We know we’re supposed to be doing customer research. We know that we should be releasing more frequently and getting feedback and having shorter cycles, lowering the complexity of our pull requests, getting into flow. But we also should be eating salads every day. Not everyone likes to eat salads every day. Just because you know something to be true doesn’t mean that is how we actually behave.”

It’s just that DevOps is no magic cure in the face of complexity within the context of these legacy organizations.

The Endless Frustration of Enterprise DevOps

About 15 years ago, someone decided to call it a DevOps transformation, evoking images of Sabrina the teenage witch spinning to change her clothes — implying it’ll be fast, easy and painless. Yet, despite thousands of DevOps consultants and coaches hired and millions spent, most traditional enterprises are no closer to feeling “transformed.” What they really feel is just a lot of frustration of trying to shove a round peg into their very square holes.

“I’m seeing a growing divide between the tech-forward companies and the companies that are going through this transformation,” Forney said. Pointing to tech giants Meta, Alphabet, Amazon and Google, she continued that “you have these tech-forward companies that are kind of leading the charge and are what the best practice measures are built upon.”

But it’s completely different for what tech had dubbed “legacy organizations.” These are like in the banking and automotive industries, which did not at first find success in building software but have more recently evolved into companies completely reliant on leveraging and building software.

“These big organizations are saying, ‘We want to go through these big transformations,’ and they’re hiring executives and leaders who have been there, done that with Google. And they bring them in and they still can’t get anything done,” she continued, which has “the transformation taking years and years and years and years and years.”

While smaller organizations can be more “agile” and get through these so-called “transformations” like cloud migrations and breaking down silos for cross-organizational collaboration relatively easily, these older organizations are still struggling — 10 years after DevOps was cool.

What causes this inertia? Forney pegs it as a combination of technical debt and legacy cultural connective tissue. “Like if you don’t stretch for a long time and you get up and try it, you won’t be able to touch your toes right away,” she said. “It just takes time. And you see a lot of turnover in leadership. Especially at these slow organizations” that are known for paperwork, bureaucracy and regulations.

Reflecting on a Fortune 100 customer, she said, “This leader just expressed severe frustration of I’m trying, but I can’t get anything done,” despite having been through successful transformations elsewhere. So they know the potential and how good it can be when DevOps is fully embraced, but they just can’t get it to work at these highly regulated 50- and 100-year-old companies.

Some say that the Big Tech layoffs of the past 18 months can be a big opportunity for these legacy organizations to attract tech talent, but, again, a Big Tech perspective may not be equipped to navigate these enterprises.

Measuring the Success of Enterprise DevOps

Part of the problem is that these more traditional enterprises have been sold on Big Tech’s developer productivity metrics, but their people, processes and often legacy technology don’t fit into these few measurements.

At the elitist of organizations, by Forney’s math, developers are spending up to 70% of their time writing and testing code, while the rest of their time is filled with meetings and context switching. But when you examine that exceptionally high 70%, she explained, you then have to consider how much time they are just “keeping the lights on” or dealing with customer support or are on call, versus “how much time they’re spending on the creation of new value.” She said it becomes a “diminishing bucket of space.”

Especially at older organizations that haven’t quite migrated to the cloud and haven’t quite moved completely from Waterfall to agile, she finds developers are often focusing on the wrong work. Or they are building workarounds on top of their technical debt as a quick win, instead of fixing with a long-term vision in mind.

“We look at organizations spending a huge amount of time doing planning and thinking these are our top priorities in the organization, but in reality, what’s going on? Are devs spending actually what you would expect to be the bulk of their time [on this]?” Forney said that “more often than not, what you see is they’re spending like 5% of their time across the entire organization level of effort on these most important things.”

This estimation comes from Uplevel’s own tooling, which pulls data from disparate sources and looks to model how time is actually spent across an organization.

“When you start to look at your entire engineering organization as an engine, you start to engineer the engineering system,” which in turn, Forney explained, allows you to focus on your biggest overall engineering bottlenecks. Only systems thinking can improve DevOps outcomes.

No SPACE for DORA?

While the seemingly popular DORA metrics and the SPACE framework are also measured at the team level and above, they simply don’t have widespread adoption in the more traditional enterprise arena.

“With SPACE, one of the biggest challenges is that it’s not super crisp [and] defined,” Forney said. The SPACE framework puts forth 25 developer productivity metrics as a starting point for the various sociotechnical things you could measure, but it’s more prescriptive around how to create context around those measurements rather than which you should use. “And so it leaves a big, wide, open-ended direction of you should measure stuff in this space, but go figure out what’s best for you. And, especially in these big organizations, people just don’t quite know what to measure.”

And then, as we previously discussed in our interview with Google’s Nathen Harvey and Michelle Irvine, there are more than four DORA metrics — more like 50 — yet everyone gets fixated on the core four: deployment frequency, lead time for changes, change failure rate and failed delivery recovery time (previously called mean time to recovery, or MTTR).

“There are a lot more than four metrics that you need to be measuring,” Forney said, that allow you to “see the narrative of balancing the people health, or making sure devs are getting enough deep work time.”

She specifically called out the need for a generative organizational culture — one grounded in information flow and trust. In fact, the 2023 State of DevOps Report added generative culture as a core capability after it found that teams with a generative culture experience a 30% higher performance, alongside a dramatic increase in productivity and job satisfaction, and a decrease in developer burnout.

Of course, especially in the typically hierarchical, traditional sectors, nurturing this level of communication and then measuring its growth is that much more challenging. The systems thinking at legacy organizations often relies on a lot of bureaucracy and a lot of conversations and intentional barriers.

It’s Time to Measure Time Spent

One reason DORA and SPACE are hard to adopt is because, even at these Big Tech companies, they are all defining and measuring DevOps success differently.

Last month, Abi Noda, CEO of the DX developer experience platform, shared the latest research into developer productivity metrics at 17 companies, all of which were originally conceived as tech companies. The vast majority weren’t using DORA or SPACE, either. He wrote, “Every company has its own tailored approach to measuring its engineering organization’s efficiency.”

Developer productivity metricss at 17 tech companies: Amplitude, Atlassian, Chime, Doordash, Etsy, GoodRX, Google, Intercom, lattice, LinkedIn, Microsoft, Notion, Peloton, Postman, Spotify, Stripe, Uber

The results of interviews with engineering leadership from 17 different tech companies, courtesy of DX.

If it’s proven that no single metric can capture developer productivity, how can anybody measure DevOps?

Forney says that Uplevel examines the metadata — nothing private or specific to an individual or team — at the engineering organizational level across many sources including:

  • Work calendars
  • Slack messages
  • Code Commits
  • JIRA and other project-tracking tools
  • Incident response tools and on-call schedule
  • CI/CD systems
  • Synchronous communication, like meetings

Particularly, she’s found that live conversations are often a way to try to sidestep bureaucracy.

“You need a true measure of how time is spent. And then you need to use that data in your planning and in your decision making about the business,” Forney explained. “This is generating alignment across your entire engineering organization so that engineering leaders are making decisions in support of what is actually happening on the ground.”

Organizations now are making top-down goals, but then, especially at these legacy businesses, she explained, “engineering teams are spending the vast majority of their time just keeping the lights on [or] on customer support issues. You name it. Some sort of thing that is just kind of keeping the whole system flowing, but is not actually adding value to the business.”

It all comes back to the original DevOps goal of creating alignment between business and engineering. The meteoric rise of platform engineering in 2023 came in part because it’s a way for business to understand what’s going on in the often-cryptic, but usually large, engineering cost center.

Understanding this flow of communication — and impeding distractions — is a good way to kick off your journey to measure DevOps in an automated way. But engineering surveys are another important source — who knows better than them what is most causing bottlenecks and breaking their flow states?

These also should include generative culture conversations to understand: “Are we collaborating in a way that is helping us truly solve customer problems?” Forney said that by combining this sentiment with data, you can begin to unlock that excitement devs feel when they are actually solving real problems:

“There are known patterns and behaviors that, if you start to measure those and try and help to balance our engineering organizations’ Triangle of Pain, you can start to influence the right conversations across your organization.”

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Postman.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.