This post is a first in a three-month series examining the challenges of Kubernetes in 2020. For January, we examine the Kubernetes developer experience.
Even as Kubernetes adoption mushrooms, it’s no secret that there are still wrinkles to iron out when it comes to developer experience — especially in an enterprise setting. “It’s probably always the case, when a new technology is very sexy and helpful for developers, that it’s adopted quickly at first, then drawbacks start to come up when you scale,” explained JF Joly, product manager at observability services provider New Relic.
For a developer who is new to Kubernetes, spinning up a single cluster on his or her laptop can seem deceptively simple. “When you start going into the public cloud, talking about thousands of nodes and thousands of clusters, the difficulty increases exponentially,” explained Ammar Naqvi, product manager at Canonical.
Developers have found a mixed bag with Kubernetes. On the one hand, it eases the provisioning and management of complex applications. But it can also quickly throw the developer into a kind of configuration hellscape, and — though this may not be the fault of Kubernetes itself — architectural chaos as different IT teams struggle to incorporate into existing operations.
So what is it really like for developers to use Kubernetes?
Different Types of Developers
The first challenge to describing developer experience with Kubernetes is that developers are not a monolithic group. The day-to-day workflow for a Java developer is not the same as for a Python developer. “We tend to say that there is no one developer experience,” explained Brian Gracely, senior director of product strategy at Red Hat. “The second thing we’ve found is that some developers just want to write software. They do not care about containers or Kubernetes.”
In larger organizations, this type of developer might not need to care. According to Joly, when New Relic moved to Kubernetes the developers were minimally impacted — it was the platform team that was responsible for managing Kubernetes. “There are a lot of people who are developing features that are not directly linked to Kubernetes, and for them the change between Mesos and Kubernetes is not very noticeable,” He said. This pattern, Joly said, is something he’s seen with many of New Relic’s customers, especially large, technically sophisticated organizations with a high degree of specialization.
In a podcast interview with The New Stack, analyst Janakiram MSV explained that one of the challenges that developers face with Kubernetes is its lack of a cohesive “platform”-like interface. By contrast, consider another infrastructure technology, the virtual machine. The VM provides a valuable service to end-users, through a set of complex technologies that all work to make a seamless, coherent experience. Kubernetes, while also capable of providing value, still hasn’t a cohesive abstraction that developers can find easy to understand.
“Kubernetes is still pretty much an infrastructure,” he said. “There is no developer experience or user experience offered to the consumers of Kubernetes.”
Originally, Kubernetes was designed for the cluster administrator. But today it has moved from being an infrastructure tool to an entire platform for managing workloads. “But the key layer, on top of the infrastructure is missing.”
Of course, there are many developers who will want and need to become proficient using Kubernetes in a production setting. Learning to use Kubernetes isn’t necessarily easy. “Most of the documentation out there is at the ‘hello world’ level,” said Mislav Stipetic, chief technology officer at Kubernetes training company Magic Sandbox. “When you’re dealing with an application that has state, that has advanced networking, that has security concerns, the only way to learn is to dig through a bunch of Medium articles.”
At the same time, there aren’t many experts out there. “We had a survey on our website that asked people to rate their Kubernetes knowledge, from one to ten,” Stipetic said. “One of the Kubernetes founders filled it out and he rated his knowledge seven or eight.”
“The challenge never stops,” Naqvi agreed. “At least that’s my experience if you’re playing around with vanilla Kubernetes on your own. The learning curve is constant.”
As developers start using Kubernetes, there’s a whole new set of jargon to learn, and understanding how everything is interconnected is often a challenge as people get started. “Another challenge we see is understanding immutable infrastructure, meaning you can’t go in and change configuration files later,” explained Roopak Parikh, chief technology officer at Platform9.
One common developer experience problem when using Vanilla Kubernetes is managing configurations. “Even if you’re an experienced developer, it can be challenging to get all the configurations right,” Naqvi explained. “It’s hard to get everything to play nice with your existing stack.”
The more pieces there are in your infrastructure — and the larger your Kubernetes deployment — the more challenging it is to manage configurations. “If you want to do something useful with your Kubernetes deployment, you’ll need more infrastructure components,” Naqvi says. And setting that upright, without bugs, is challenging.
In vanilla Kubernetes, many new users don’t understand how to integrate Kubernetes with a CI/CD system. This is something most of the managed services provide — and, according to Joly, a huge benefit for many of New Relic’s customers. “There is really a huge uptake in our customers for managed services like GKE or EKS,” he said. “It really lowers the barriers.”
In some ways, the Kubernetes’ challenges stem from the lightning-fast adoption. Joly’s seen many cases where companies haven’t taken the time to have an architect establish policies and create a blueprint for how the organization will standardize on Kubernetes, leading to chaos as multiple ‘rogue’ Kubernetes deployments are cobbled together. It’s also sometimes tempting for new developers to rush in without a clear understanding of the basic building blocks that make Kubernetes run. All of the layers of abstraction in Kubernetes are essential to operating at scale but can seem like a distraction to a developer who just wants to test something quickly.
Another common pain point is visibility into the system, or, on a more basic level, how everything fits together. “We found this out when we started with Kubernetes internally, as well as without customers,” Joly said. “Understanding how things are connected was a big challenge. So, if I’m having an issue on my node, which instance of my application is it impacting?”
On the other hand, one of the biggest learning curves related to Kubernetes adoption might be organizational. If instead of building a new platform team to manage Kubernetes the company simply takes the existing infrastructure team and tells it to be responsible for this new infrastructure, it goes against everything that team has been measured on to date — stability and high availability, which means few changes. “Where we typically see success is the company says, you know, Kubernetes is aligned to the idea that we want to build newer applications that move more frequently,” Gracely said. “So we should design around that idea that environments are going to change frequently.”
It can feel like a thankless slog to some developers, too. “You’re not even awarded for this transition, it’s kind of invisible work,” Stipetic said. “The people, the knowledge silos have to change, it’s a lot of effort before you see results.”
The fact that Kubernetes has been adopted so quickly points to something that came up repeatedly in discussions of developer experience: It’s worth it. “Kubernetes makes easy things hard and hard things possible,” is how Stipetic puts it. Once a developer is familiar with Kubernetes, the advanced mechanisms can make life a lot easier. It also creates a standard way for an entire organization to handle container orchestration, which makes it easier to onboard new team members.
“Kubernetes really provides value that you didn’t have before, in terms of orchestration, better usage of resources and faster time to market,” Joly said.
New Relic and Red Hat are sponsors of The New Stack.