How Platform Engineering Can Improve DevOps Collaboration
You don’t have to be an industry analyst to know that platform engineering is a pretty hot topic right now; it’s arguably the topic of the moment in software engineering. In fact, our new survey of New Stack readers revealed a keen interest in it; our audience ranked it fourth among topics they want to learn about in 2023.
But the level of interest should also be a sign that whatever platform engineering has set out to fix isn’t yet fixed. If it’s hot, that’s because it’s causing a stir, not because it has matured and settled into established mainstream practice.
At any rate, the mainstream is looking like platform engineering’s ultimate destination. According to Gartner’s forecast, 80% of software engineering companies will have platform engineering teams in place by 2026.
And the current uncertainty in the global economy is already ushering in a round of belt-tightening by organizations and their IT departments, putting more pressure than ever on developers to build and deploy more efficiently.
So why isn’t platform engineering the status quo yet? Does the fact that we’re all talking about platform engineering mean DevOps never really repaired the relationship between developers and operations teams? And if so, how can platform engineering help heal that relationship and make it more productive?
The Limits of DevOps
DevOps was much more than just being a new way of delivering software. It also brought with it a completely new way of thinking about your role as an individual within a team and how you interact with those around you. Amazon Chief Technology Officer Werner Vogels’ dictum “you build it, you run it” emerged on the scene as a whole new way of thinking.
That was then, however. The world has changed and with it, the idealism of the 2000s now looks dated, unsuited for a time of increasingly complex and highly distributed software systems.
“I don’t think he really understood how complex the world was going to be,” Kaspar von Grünberg, founder and CEO of platform engineering company Humanitec, told The New Stack. “If you’re looking at what we’re deploying today, it’s a step function more complex — you have dozens of microservices and tools; we have a global scale and much more speed. We’re deploying all the time. It’s so wildly complex.”
This increased complexity has created new challenges in how we build and deploy software. From a team and organizational perspective, that leads to new stresses and strains for the people actually doing the work.
Disciplines like site reliability engineering may help in managing complexity and handling concerns that go beyond the scope of a development team’s work, but they don’t solve the fundamental issue that individuals are now having to do more with more — more tools, more complexity, more decisions to make every single day.
Lee Diatiangkin, a product manager at IBM who has been working in the platform space for the last decade, sees the issues as being caused by two things. Either, Diatiangkin told The New Stack, “organizations try to shift as many DevOps tasks as possible left to the developers, which puts a high cognitive load on them, or they reflag their Ops department with fancy new names but keep their ticket-based workflows, leading to frustration on all sides.”
Von Grünberg echoed this assessment, saying that managers too often abuse the DevOps principles of cross-functional teams and collaboration, turning it into “everyone does everything.”
He elaborated: “What that actually leads to is large-scale burnout because you’re asking so much from people. You’re hiring them to be the best at ReactJS, but you’re also giving them 20 other things that they’re supposed to do, and that’s very, very unproductive and it leads to things like ShadowOps.”
Part of the problem, von Grünberg suggested, is that “specialization” is now seen as a dirty word. “If you have a very complex toolchain, specialization isn’t something bad,” he said. “Specialization doesn’t mean you have silos.”
As Ditiangkin put it, “the walls have been removed, but now there are blurry lines on who is responsible for what.”
The question, then, is how we can make those lines clearer? How can we enable specialization in a way that simply allows people to get on with what they’re good at? At the moment industry consensus seems clear: the solution is platform engineering.
What is platform engineering? It’s “the discipline of designing and building toolchains and workflows that enable self-service capabilities for software engineering organizations in the cloud native era,” Luca Galante, product lead at Humanitec, wrote in a blog post on the Platform Engineering website.
“Platform engineers provide an integrated product most often referred to as an ‘Internal Developer Platform’ covering the operational necessities of the entire life cycle of an application.”
How does platform engineering make the lines between Devs and Ops clearer? Platform engineering builds the IDP, Ops can configure infrastructure and focus on real operations and infrastructure problems. Meanwhile, developers are enabled to self-serve, so they don’t need to bother Ops anymore. And Ops is no longer annoyed by repetitive requests from developers.
Platform Engineering’s Growing Popularity
Much was made of the importance of digital transformation during the pandemic, but even as much of the world appears to be exiting from that difficult time, the ability to move fast without breaking things — thus building trust between Devs and Ops — is becoming particularly important. True, that might have always been the case; but in 2023 margins (literal and metaphorical) are much finer.
This environment is a fertile ground for platform engineering to grow. The combination of increased technical complexity and challenging times for businesses means that improving the ways in which different technology teams work together, making things simpler and more efficient, is essential.
What’s more is that there are clear signs that the practice of platform engineering is, indeed, growing. For instance, 2022 saw the first-ever PlatformCon, signaling that there is not only a thriving community out there but also a need for a space where it can come together to evolve and better understand the discipline. (The call for proposals for the 2023 event is open until February 28.)
Common Misconceptions about Platform Engineering
If we follow analysts like Gartner we could make the mistake of thinking that platform engineering is a destination that the industry is hurtling towards. The reality, however, is that platform and tooling considerations are already here, a common part of many engineers’ day-to-day work.
As von Grünberg put it, “There’s an argument to be made that you’re already building an internal developer platform — if you don’t build it, it builds itself.”
Thus the question organizations should be asking themselves is not so much, “Should we build an internal developer platform?” but rather, “which workflows and tools do we already have in place?”
Platform engineering, then, allows organizations to be deliberate in how they support and enable development teams by building an Internal Developer Platform.
Even if you don’t purchase an existing portal — or adopt an open source solution — such is their visibility in the industry that they have arguably encouraged lazy thinking when it comes to developer effectiveness.
“Many, many platform engineering teams start with building a fancy UI like a developer portal on top of something,” von Grünberg said. This is, he suggested, the wrong approach as it will likely cost significant time and money with little impact. Developer portals or service catalogs usually focus on Day Zero scenarios, like creating a new service from a template. “You’re optimizing something that happens 10 times a year. How much more efficient do you make it?”
Von Grünberg is eager to make the distinction between developer portals and internal developer platforms clear: “A developer portal is not an internal developer platform.” This is a point that has been made by others across the industry. Gartner, for example, has offered a clear definition: “Internal developer portals serve as the interface through which developers can discover and access internal developer platform capabilities.”
Precisely because a developer portal is an interface, it can capture organizational attention in a way that is a little more difficult for an internal developer platform. A portal, after all, is a visual thing — you can see it; it will probably have a certain aesthetic quality. But the most important value that platform engineering and internal developer platforms provide is making developers more productive.
A portal is only the interface of the platform, but not the platform itself. If you build the interface before you get the platform architecture itself right, it is like building a house’s front door before creating its foundation.
Taking a Product Management Approach
Instead, as Ditiangkin emphasized, you should take a product approach to your platform, as it is really about “a product mindset that sees developers as customers.”
In other words, it needs to be treated in exactly the same way as you would treat a product being built for an external market. This isn’t just a mantra; there are a number of practical steps such an approach necessitates. As Ditiangkin outlined, “You must conduct user research before building your platform, get stakeholder buy-in, and evangelize the platform to developers to ensure adoption.”
A product management approach enables platform teams to be sensitive to the specific and unique needs of developers in their organization; it allows them to go deeper than a few cosmetic changes. If anything can do the work that DevOps was supposed to do when it was first coined nearly 20 years ago, this shift toward product thinking might be it.
This isn’t easy: von Grünberg stressed that, unlike developer portals “there is no internal developer platform out of the box” and that they need to be built according to the demands of your specific context. However, there are nevertheless two key elements that can serve as a guide.
The first is the notion of a “golden path.” The term refers to paths that provide a clear route but in a way that is unobtrusive, balancing guardrails with autonomy. “We do not want to impose abstractions; we want to make sure people can self-select the level of complexity they feel comfortable with,” von Grünberg said.
The second is what he called “standardization by design.” This ensures greater efficiency and consistency is built into the way every developer works by virtue of it being embedded in the platform. It helps, for instance, you to minimize and ultimately reduce the number of configuration files that are being managed across your teams.
Putting Platform Engineering into Practice
The principles and theory behind platform engineering are one thing: where it really matters is how it’s put into practice. Von Grünberg advised platform teams to start small and focus on what he calls the “lowest common tech denominator” — what’s used across all teams.
What does that mean? “You will not be able to build a platform for everything. And yes, it’s probably about containers and K8s.”
Then, within your engineering organization which might have dozens or even hundreds of Dev teams, it is important to identify your team of future evangelists. You should select one team of innovators, that works on an app exactly with your target tech stack and that is really willing to be your first customer. Developers equal customers of the platform engineering team, using the product, the platform.
And then constantly gather feedback from them and iterate on the platform based on this feedback.
This will help ensure that you get that buy-in within your organization; you essentially create what von Grünberg calls “hardcore fans” who will evangelize for the great work you’re doing as a platform team.
Getting this buy-in is important for the simple reason that building an internal developer platform is difficult. On the one hand, this is due to the technical challenges; von Grünberg says the process can sometimes be “very manual and tedious,” such is the complexity you have to contend with to simplify and standardize infrastructure configurations and tooling.
This is partly where Humanitec comes in; its more than 150 open source drivers, for example, can help simplify a huge range of integrations, while its platform-agnostic workload specification Score, von Grünberg said, can help devs to “specify workloads in a developer-centric and platform-agnostic way.”
While these sorts of tools may go some way in helping to make platform engineering more accessible to organizations without the resources of the likes of Google or Netflix, is only part of the picture. It’s also about continuing to foster a sense of community in which people identify with platform engineering as a discipline and feel motivated to talk to each other about it, share experiences and ideas.
Community and collaboration are a core part of Humanitec’s philosophy — it’s why, for example, Score and Humanitec Drivers are open source (Score was open sourced in November, acquiring more than 7,000 stars and 2,500 forks on GitHub in its first three months). It’s also why Humanitec is one of the organizers of PlatformCon; the company is keen to support and foster a community, helping organizations of all kinds to adopt platform engineering.
Making Platform Engineering Mainstream
So how does platform engineering become a new normal inside the industry? There are still many steps to get to that; talent, for example, remains a huge issue (Von Grünberg claimed he “can name 20 organizations right now that are looking for a platform product owner.”)
But as the community grows and knowledge is better distributed, the platform engineering role will likely lose its mystique as people come to view it as a viable career option with its own set of capabilities and skills. Similarly, organizations will need to adapt to shifting structures and new ways of thinking about hierarchies. However, this, again, is about education — like an internal developer platform, the practice itself needs evangelists and communicators.
Platform engineering’s success in the years to come could be viewed in how it reorients the relationship between engineering teams. Can it, to use Ditiangkin’s words, help “regain a clear separation of concerns without putting Devs and Ops back into their silos”? There’s certainly a lot at stake — the industry needs to get it right for the sake of everyone building and deploying software.
Want to learn more about platform engineering? Check out this episode of The New Stack Makers podcast, with guest Kaspar von Grünberg of Humanitec.
