Measure Developer Joy, Not Productivity, Says Atlassian Lead
You’re measuring your developer productivity wrong, according to Andrew Boyagi, who leads the Agile and DevOps evangelism team at Atlassian.
There’s no doubt the last year, which has seen a tightening of teams and budgets, sparked a refocus on making developers more productive. Yes, the tech industry has always tried to measure developer productivity, but organizations have often fallen short by trying to benchmark themselves against others.
Also notable, 2023’s push to increase developer productivity seems to lack the prior years’ concern about developer burnout and well-being.
Which is a wrong move. The 9th Annual State of DevOps Report, for which Google’s DevOps Research and Assessment (DORA) team interviewed nearly 3,000 tech workers, was released in October. This year’s report looked to measure three outcomes:
- Organizational performance: generating value for customers and community.
- Team performance: empowering teams to innovate and collaborate.
- Employee well-being: reducing burnout and increasing satisfaction and/or productivity.
The result was clear: Culture is everything. In particular, a high-performing organization that cultivates a generative organizational culture — one grounded in high trust and information flow — experiences 30% higher performance, with a substantial increase in productivity and job satisfaction, alongside a decrease in burnout, according to the report.
It’s been long proven that happy employees are more productive, which is why Boyagi argued you should focus on developer experience (DevEx), not productivity. Here’s what that means and how to figure out how to do it in your organization.
Why Measure Developer Productivity at All?
“A lot of the senior leaders I speak to from other companies, they’re so focused on measuring developer productivity, but not many of them talk about actually helping developers be productive,” Boyagi told The New Stack
Engineering is inherently a science, so we know we can’t improve what we don’t measure. But what we measure has to matter.
“More than any other craft, developers love their job,” he said, which is emphasized by their mentoring of other devs and sharing continuous learning on YouTube, Discord and LinkedIn, and by the 413 million open source contributions made last year alone.
“And it’s not like they don’t want to be productive,” he added. “So you got this scenario where they love their job, they want to be productive, they don’t feel productive, and then leaders want to measure productivity.”
So, if happier developers are more productive, Boyagi argues, then developer productivity becomes a byproduct of developer joy, which is the combination of:
- Developer experience: how engineers feel about their tools and frameworks.
- Engineering culture: how they get work done and the leadership supports them — an organization’s unique personality.
Even the Atlassian CTO Rajeev Rajan has argued developer joy is the key to unlocking developer productivity. That joy, he wrote in a company blog post, is measured in a three-part framework:
- Provide awesome tools.
- Empower teams.
- Foster an amazing engineering culture.
With this level of uniqueness being so important, Boyagi said, you can buy the same tools as a high-performing company. But you can’t “do the same as them because it’s just not going to work. In fact, it’ll probably make things worse if you try and do that.”
“You will not get one set of metrics that you can universally apply to every organization because productivity is a byproduct of developer experience and engineering culture. And they’re so unique to an organization that it’s not possible just to take a standard set of metrics and apply them.”
— Andrew Boyagi, Atlassian
So, when senior leadership is under pressure to show the outcome of one of their most sizable operating expenses, what’s a tech company to do?
First, Boyagi suggested, change your questions. Instead of “How do I increase developer productivity?” or “How can I measure developer productivity?” try “How can I make developers happier?” and “How can I help developers be more productive?”
The questions can help steer the conversation in a more useful direction: “I think every company has to go on a journey and do what’s right for them in terms of productivity. But I don’t think I think measurement is the thing we should be talking about.”
First, because productivity for knowledge workers has always been one of the hardest things to measure. And, he added, because we need to take inspiration from other companies, not replicate what they do.
Also Read: Developer Empowerment Via Platform Engineering, Self-Service Tooling
How Atlassian Measures Developer Experience
Boyagi doesn’t suggest you try to do what Atlassian does. But feel free to take inspiration from and leverage its DevEx strategy, as well as those from the likes of other high-performing organizations like Google, Netflix, LinkedIn and Spotify.
Atlassian offers five-minute anonymous surveys for each engineer once a month, as well as weekly team CheckOps. CheckOps is like a retrospective — What was good? What could’ve been better? How can we improve? — within the Atlassian developer experience platform Compass. Each team goes through its scorecards and statistics within Compass to reflect on the week. Also, each team reaches an internal consensus to answer one particular question: How does the team feel this week?
CheckOps are for each team to reflect on the week, while the anonymous engineer surveys offer a pulse for the whole organization.
Atlassian also, Boyagi said, “takes inspiration” from DORA (change failure rate, failed deployment recovery time, deployment frequency, and lead time) and SPACE metrics (sociotechnical factors that influence DevEx). Despite a common misconception, neither are developer productivity metrics, he said, but rather they are different angles that Atlassian factors into measuring the company’s whole developer experience.
Interestingly, story points — an estimate of the overall effort needed to complete a piece of work, which is found among the suggested SPACE metrics — are among the software development metrics most associated with Atlassian’s product Jira, and perhaps the most controversial and easily gamified.
“Story points are an estimation too,” Boyagi said. “In my opinion, they are not a productivity measure. But, like anything, there is a scenario where story points are a valid estimation tool. It depends on the organization, it depends on the scenario. There is no metric I can say sit here and say to you, ‘Everyone should do this,’ because nothing really applies universally.”
One metric Atlassian finds particularly useful is how much time developers spend looking for documentation. At most engineering organizations, finding stuff is a huge source of frustration that breaks developer flow state.
In fact, according to last year’s Stack Overflow Developer Survey, 62% of all respondents said they spent more than 30 minutes a day searching for answers or solutions to problems; 25% spent more than an hour. In a team of 50 engineers, that equals well over 300 hours of “lost time” a week.
“There’s been a shift to cloud. And there’s so much that developers have to do now. And the way that we see it is developers need a single tool to navigate all the information that they’ve got and collaborate across their tech stack,” Boyagi said. “It’s not just a tech problem. If it was just a tool that could solve everything, then that’s easy, but it’s technology [and] it’s a collaboration problem as well.”
Atlassian first built Compass as a way to get back that lost time and to decrease cognitive load. Via one URL, it helped solve common organizational mysteries like who owns what component, what it does, which are the up-and downstream dependencies, and what documentation is needed.
Compass now has a 90% internal adoption rate; in October, it was released for general availability to the public.
“All the information in Compass is stored in a standardized place,” Boyagi said. “And it really empowers developers to work autonomously. It increases their velocity.”
How Atlassian Continues to Impact DevEx
Seventy-five percent of the Fortune 500, and about 125,000 of tech companies, use Atlassian’s developer products. And yet, Atlassian doesn’t use its tools to measure developer experience at its customers, Boyagi said: “Developer experience is quite unique to the organization It’s not something that someone else can measure for you.”
Instead, the Atlassian team gathers customer feedback and then provides tooling to enable its customers to improve their own developer experience.
Atlassian’s developer experience platform Compass, integrates with developer staples like GitHub, GitLab, Jira, Bitbucket, CircleCI, LaunchDarkly, Snyk and PagerDuty to gather metrics around developers’ ability to put ideas into production, measuring if:
- The platform is easy to build in.
- Pull request (PR) cycle time is low.
- The number of open PRs is not too high.
- Build times are short for faster development loops.
- Unit test coverage is high.
One of the main drivers of this year’s popularity around platform engineering is developers’ demand for extensibility. With that in mind, Compass is also built on an open toolchain and public APIs, so each organization or engineering team can make it suit their own unique developer experience. Other organizations are building their own apps on the platform.
“It negates the need for software teams to go and look in 30 other tools to find what they need,” Boyagi said.
He gave the example of the developers needing to use a monitoring or logging tool to check on the health of their application.
“Just to do that, a developer has to have a login for the tool, know how to navigate that tool, know how that all works, find the software component that they’re looking for in that tool, so navigate the hierarchy,” he said. “They find what they’re looking for. By the time they’ve done all that they see what they needed, and they go switch back to another screen to work out: What was I looking at in the first place? Why do I need that information?”
Instead of all this context switching, Compass allows orgs to plug all that data in, and developers can look for their software component right there, reducing cognitive load and improving the developer experience.
Atlassian encourages its engineers to dedicate 10% of their time to address things that make their jobs worse and improve them. This can include extending their internal instance of Compass, which then can help make other teams’ lives easier too.
Another welcome use case for Compass customers like Boden, DropBox, and KFC UK & Ireland, is governance on demand. Currently, most organizations run governance via meetings, where, Boyagi said, engineering teams must prove how they’ve met standards before getting approval to release.
“It doesn’t ensure a quality outcome to production,” he said. “It’s slow. And it’s frustrating for everyone involved.”
Instead, Compass includes scorecards, which governance teams can use to distribute their requirements, like service readiness. This means that everyone is clear up front on the requirements, which are automatically updated.
“The best part is, you don’t actually have to go to a meeting anymore,” Boyagi said. “Because in those meetings, it’s not helpful usually explaining what you’ve done to someone who’s never delivered software. You can say anything, they don’t really understand what you’re saying anyway, right?”
Instead, governance teams can filter the scorecards, Boyagi continued, based on whether developer teams are failing or about to fail in meeting the requirements. Then they can reach out to that team and help them get back on track. “That’s a very different scenario,” he said, “than ‘come and show me what you’ve done.’”
By helping engineering teams feel like everyone wants them to succeed, you’re able to increase developer joy and improve developer experience.
