TNS
VOXPOP
Where are you using WebAssembly?
Wasm promises to let developers build once and run anywhere. Are you using it yet?
At work, for production apps
0%
At work, but not for production apps
0%
I don’t use WebAssembly but expect to when the technology matures
0%
I have no plans to use WebAssembly
0%
No plans and I get mad whenever I see the buzzword
0%
Platform Engineering

GitHub Developer Productivity at 30 Billion Messages per Day

GitHub is uniquely positioned to impact developer experience at a global scale. So how does GitHub measure developer productivity? We chat with Akshaya Aradhya, senior director of engineering at GitHub, to learn how her team works to increase productivity across the tech company
Dec 11th, 2023 3:00am by
Featued image for: GitHub Developer Productivity at 30 Billion Messages per Day

If GitHub is used by 85% of software engineers, developer experience at GitHub becomes a nesting doll of developer productivity. The world’s largest repository — which alone features 142 million users of the six-month-old AI pair programmer Copilot — has a unique ability to impact developer experience globally.

This scale also complicates things. GitHub features a service-level agreement with 99.9% uptime serving users across nearly 400 million repositories. This has the world’s largest software development platform supporting a staggering 40 billion messages a day — yes, that’s with a b.

In our ongoing journey to unlock developer productivity across the software development lifecycle, The New Stack sat down with Akshaya Aradhya, senior director of engineering at GitHub, to learn how her team works to increase productivity across the tech company — which in turn should increase productivity across the vast majority of tech organizations.

GitHub’s Platform Engineering Organization

Her team is part of the infrastructure and platform engineering organization within GitHub — which horizontally supports the whole company, including:

  • Application teams
  • Code search
  • GitHub Actions
  • Copilot
  • GitHub Next R&D team
  • Upcoming partnerships with Azure Cloud

The platform team is made up of almost 300 GitHubbers, while Aradhya leads about a third of that. The platform team also partners heavily with the other major horizontal entity — security — in everything that they do.

Her organization used to be called data services and is responsible for what she refers to as the traditional problems with data, including the data pipelines, the data side of operations, and messaging queues.

“We have Hydra and Aqueduct which processes billions of messages per day — 30 billion messages or something like that,” she explained, but, “if you include the developer ecosystem on top, it’s even more.”

Her about 120-person team also tackles data storage problems — object storage, cold storage, blob storage — across all of GitHub, both internally and externally — as well as data patterns, next-generation scaling issues, productivity issues and more. Aradhya told The New Stack that her team focuses on: “How do we truly listen to the developers, and then talk with our own application, test with our own data, learn, and then come up with products?”

And then of course they have to measure it all toward continuous improvement.

“How do you improve developer effectiveness and developer productivity, and still ensure seamless deployment, still make it very easy for people to interpret the code as much as possible, and [to] automate things that people will love, especially when you’re running distributed teams across different time zones?” This all has Aradhya’s team obsessed with enabling their highly global team of developers to overcome their most frustrating bottlenecks. “Why do I need to wait till the people from Germany on Amsterdam wake up? I’m just blocked.”

With GitHub being a remote-first company since well before the pandemic, its thousands of engineers make for great first customers and continuous feedback, like with onboarding.

“When you entered the software industry, you are told that ‘Oh, XYZ is a team lead’ or ‘This is a senior engineer, if you don’t understand the code to ask them.’ And there’s a waiting period to find that person, to explain the code to you for knowledge transfer, and the awkwardness that ensues if you want to ask the question again, because you forgot,” Aradhya said, which is awkward for anyone, especially a newbie, but downright uninclusive for many.

“There’s a lot of neurodiverse communities that are like ‘That’s a problem with my executive function skills. I hope I’m not judged based on that.’”

With this sort of feedback — and likely the personal experience of the 30% of GitHub employees who identify as neurodivergent — the team has built a feature that, when you select a code snippet, Copilot’s chat feature — think Clippy on steroids — will explain how the code works.

“Across any ID. No matter what you’re working on. Your Copilot will suggest methods and will tell you how the code works,” she explained, like making sure you aren’t writing the wrong method name from a different class. This helps increase developer confidence, alongside the speed of writing code, while decreasing the probability of introducing bugs.

This innovation in response to internal developer feedback is part of the reason why Copilot adoption exceeded all expectations, with about 92% of GitHub users in the U.S. already using it since it was released publicly last June.

It’s also the result of GitHub dogfooding everything, acting as the first customers and testers.

Extreme Dogfooding of Developer Experience

“Before we release anything to GA — or even alpha, beta versions of anything — we want to make sure that it’s developer ready,” she said. This feeds into the GitHub values, which are all around fostering collaboration and belonging, which is driven by continuous feedback.

“When we collaborate, we always ask our questions related to productivity, like: What do you think AI’s impact [is] on your developer experience? We use that information to tie that to what will the enterprise think. How do we want to help an enterprise company — whether it’s Apple, Walmart, Tesla, Google — understand and support engineering teams?”

Like many of the other organizations we’ve spoken to about developer productivity, a lot of this feedback is gathered via developer surveys.

“When we talk about developer experience, people tend to traditionally link it to the product or the experience like the UI. It’s so much more than that,” Aradhya explained. “If you think about it mathematically, it’s developer productivity plus the impact developers produce plus developer satisfaction. If you sum that up and add an exponent, the developer experience is all that and so much more.”

The GitHub research teams then publish internal blogs and, early on, their DevEx insights are involved in product-market fit discussions and in senior leadership meetings.

One of the pain points GitHub research consistently identified early on was waiting for tests to complete or for builds to finish — Aradhya calls it death by waiting for collaboration: “How do you talk to people or know what everybody’s working on? How do you automate reviews? How do you make builds go faster? How do you take care of bugs?”

That is where AI will happen. In fact, while a lot of people are boasting early gains in individual developer productivity, faster incident response and error reduction, one of the most surprising results of the 2023 GitHub Octoverse Report — which annually surveys anyone with a GitHub account — found that 81% of those surveyed expect for AI to increase team-based collaboration.

How to Measure Collaboration Metrics

Internal surveys echoed this, where GitHub developers requested that their collaboration be measured more — because you can’t improve what you can’t measure. But how can you measure human interaction? This goes well past the bare minimum of Slack, Jira, pull requests and docs, Aradhya said. In the developer productivity world, this examination of collaboration improvement factors in a breadth of synchronous and asynchronous communication. They can get significant quantitative metrics from these tools and messaging queues. Her team also focuses on the platform engineering goal of discoverability:

  • How do you know how to discover something?
  • How do you measure the flow of information between two people or teams?
  • Are you working on the right problems?
  • Are you attending the right meetings?
  • Are you able to protect your time so you can reach that developer flow state?

At this remote-first company, a lot of the onus is on the individual to understand how they work best and are able to self-organize their blocks of focus time. It’s not surprising that GitHub would have a higher percentage of neurodivergent teammates as this is a very inclusive way of working. GitHub also provides unconscious bias, privilege, and allyship training for all employees.

Collaboration metrics at GitHub intentionally include diversity, equity, inclusion and belonging. Like most tech companies, they have had problems, Aradhya mentioned, with the mentoring of women and other under-represented populations in tech.

“We had challenges with mentorship sponsorship [and] mentor-mentee relationships,” she said. Not just in the setting up of formal mentorship programs, but “we also had problems where people thought they were their mentors, but they didn’t produce enough value for the mentee, or the mentee did not recognize them as someone who could mentor them.” This negative experience disproportionately affects those from under-represented groups.

In response to this feedback, they’ve set up more formal mentoring processes with training.

Measuring Developer Experience at GitHub

The massive Octoverse Report is just one annual survey among many sources of internal and external data. While teams are able to get qualitative metrics like the famous core four DORA metrics, there is usually a mix of quantitative and qualitative surveys throughout any developer productivity engineering journey, including the SPACE framework and DevEx metrics.

GitHub R&D are rapidly testing proofs of concept multiple times a week, Aradhya said, for nearly instant feedback. The platform and infrastructure team run different developer experience surveys every few weeks all the way to once a quarter. Never a three-page survey that no one will want to answer, she assured, but rather a mix of shorter ones that have higher response rates.

“Not everything needs to be a survey,” she continued. For example, product teams are constantly taking action from customer conversations, and have quick rating system feedback during events.

“I even remember at an engineering leadership conference, I was standing in line for the food trucks at a lunch break, and then somebody saw my GitHub jacket and they were like, ‘What do you work on?’ I said that we support XYZ and Copilot. And the second I said that they started giving me feature requests,” she recalled. “By the time the line of the food truck ended, I had 18 different opinions, different products that we could enhance or improve,” which she immediately fed back to the respective product teams.

GitHub is a very feedback-driven company. Aradhya said developer productivity and developer experience measurements are a constant topic at senior leadership meetings.

At GitHub, these productivity metrics include:

  • How much time does it take to get feedback from users?
  • What does asynchronous communication look like?
  • How much time are you able to work in blocks without meetings or interruptions?
  • How much time are you spending building solutions to novel problems?
  • How much time do you spend writing code or fixing bugs?
  • How often do you encounter security or vulnerability issues?
  • During new product releases or updates, how much time are you spending on upskilling yourself? On the mentoring and upskilling of others?
  • How much time is spent writing automated tests?
  • How often are you switching contexts?
  • How productive do you feel?

Many of these smaller internal metrics tie into the bigger ones, she explained. For example, when Aradhya was an engineer, she realized it’d be quicker in the short term to just run a test in production herself, but in the long run, it’s a waste of time not to write a repeatable, automated test. Automating out repetitive work is an important goal of developer productivity engineering.

With her team leads, she said, “I take an inventory of the number of human touch points something has.” It’s also important to measure how much time developers get to spend innovating over unique problems, which, she says, is key to retention. “This is designing for novel metrics — novel communication, new APIs, for all the latest problems. Basically, do you have the 10x mindset, do you understand the customer problem, and what are the ways you would like to solve it?”

Global Scale Demands Unconventional Developer Productivity

These are what Aradhya calls the “unconventional developer productivity” stuff that her team looks to dogfood solutions over. GitHub works to use these metrics to cultivate that 10x developer mindset, which she defines as “a developer who has a vision and technical competency to envision what problems they may encounter in the long term as well as short term, and thinks about issues in terms of scaling users beyond product-defined boundaries or engineering leadership defined boundaries.”

This consideration of scale was crucial to the success of Copilot, as they predicted it would have maybe 15 million users at the start, but it quickly became clear that 115 million would be a closer estimate — when it would be too late to start worrying about availability, reliability and security.

Anything GitHub rolls out simply has to scale.

“A 10x developer could foresee that and then provision backups, provision Part B, help the tech strategy so that we can know the unknown unknowns. And then figure out a way to scale based on the consumer demand without compromising code quality,” Aradhya said.

In general, at Microsoft, which owns GitHub, she said that anyone can have an opportunity to be a 10x developer, regardless of level, by bringing out-of-the-box thinking, and an emphasis on company culture and collaboration. She gave the examples of, “How did you scale up the team around you? How did you impact them? Did you change the team in a positive way or change the company culture in a positive way?”

Another developer productivity metric area that GitHub focuses on is of course the tooling, such as looking to answer:

  • What does it mean to have a fully configured environment?
  • How easy is it to build on top of that platform environment?

GitHub also gathers a lot of automated feedback, looking to always tighten feedback loops, like from validation and compliance checks, to helping developers understand where they are most unproductive, and especially getting customer feedback to the developers, making sure they are incorporating feature requests.

All this is measured with an ongoing mix of anonymous and authenticated surveys, issues, pull requests, and more.

“If you inundate people with surveys, they stop answering so we get creative here,” she said, “And also we look at our own analytics and funnels that we operate internally.”

All of these results are communicated with the rest of the organization, explaining:

“This is the percentage of time you’ve spent on X. And we improve that with Y. And then ask for feedback again.” She explained that anyone can internally see this information, all the engineers across the world.

Group Created with Sketch.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.