What Is Platform Engineering and When Should You Invest in It?
As application platforms grow larger, the idea of DevOps teams where developers support the software development life cycle, but also manage infrastructure and the platform, is beginning to reach the limits of what these teams can support. Rather than making their best application developers work on infrastructure problems, more organizations are realizing that a centralized platform team that specializes in that area is a better use of their developers’ skills.
Platform Engineering: What Is It and How Did It Come About?
Platform engineering is essentially building (selecting/standardizing on), operating and managing the infrastructure that supports first- and third-party applications. In the days before cloud native application development, a central team provided compute infrastructure for enterprise developers to build and host their applications.
At a certain point, those developers moved to a microservices-based architecture. They didn’t just need virtual machines or servers where they could run their applications. They were building those applications in containers, using Docker for their development infrastructure, and then using some form of orchestrator to run their applications. What they needed was a development platform for containerized application hosting.
By that time, Kubernetes had become the dominant container platform for orchestration. There were others, like D2iq and Docker, but Kubernetes ultimately won the race. The central teams started providing a development platform that included the Kubernetes orchestrator and everything else enterprise developers would need to build and host their applications.
Today, I’ve seen organizations include things like the repo structure (registries) and any kind of application operations infrastructure (build infrastructure, monitoring infrastructure, data stores, logging infrastructure, etc.) in that platform.
I’ve also seen organizations use application-layer infrastructure as part of their platform, including CI/CD pipelines and service mesh. This sometimes includes capabilities around blue-green deployments and canary deployments included as part of the platform. But at its core, the platform is compute infrastructure that’s aligned with a microservices architecture in a true multitenant fashion. That’s what platform teams are responsible for managing.
Platform Engineering vs. DevOps
In many cases, platform engineering is an overlay to DevOps. While DevOps teams tend to work more closely with application teams to understand their operational requirements, platform engineering’s charter becomes standardizing on solutions and infrastructure to support the entire estate (such as choosing a consistent data plane). Overall, platform engineering simplifies operations and reduces overhead to standardize on infrastructure components.
With organizations including so many types of infrastructure as part of their platforms, it makes sense for a centralized team to manage it all. If you have hundreds of developers in your organization, it doesn’t make sense to have them all doing the same set of things.
It would be an extreme waste of resources if every Dev team had to manage their own logging, registry and operating infrastructure. More importantly, it is difficult to implement a corporatewide regulatory framework for security across the platform if every team is doing their own thing. You need a centralized platform, and a specialized team — the platform engineering team — to manage it.
If we look at DevOps as a setup, you can never expect a true full-stack software developer to also own systems and infrastructure — this requires a different skill set, mindset and approach. It’s not realistic to have software developers who are writing application code also write platform code or platform automation scripts and manage operations.
When I have seen DevOps teams owning everything end-to-end, there has always been a specific person on the team with operational skills who was owning the Ops aspect of the DevOps. In all other cases, it just doesn’t work.
When to Invest in Platform Engineering
If your organization is entirely in the cloud, and your Dev teams are running their applications on managed Kubernetes services, then a large part of your compute infrastructure and platform is managed by the service provider. In this case, you wouldn’t necessarily need to centralize that function, but there would be bits and pieces of that platform that you would need to run, operate and manage. That’s where it would be useful to invest in platform engineering.
For example, let’s look at logging across all Amazon Elastic Kubernetes Service (EKS) and Azure Kubernetes Service (AKS) clusters in an organization. If you have teams that are building their own Kubernetes clusters through open source distros, you would absolutely want to centralize that function. Why? Primarily because managing, building and operating these different distros is not straightforward; it requires a specific set of skills, and there is quite a learning curve. You certainly wouldn’t want every Dev team in your organization to have to overcome that learning curve.
Also, without a centralized logging infrastructure and a centralized team to oversee it, you run the risk of inefficiency as multiple teams work toward disparate solutions to the same problem. Last, having a platform engineering team in this case would enable application developers to focus on innovation and delivering new features that add value for the business.
An even more important reason to invest in platform engineering is that if you are building everything yourself, you are responsible for managing security, compliance and uptime for your infrastructure. There are specific requirements and playbooks for operating OS distros securely; you don’t want to have to do this work N number of times with every single team in your organization. This is when you want to invest in a platform engineering team.
The main takeaway here is that even if your organization is entirely in the cloud, at some point you will likely want to invest in platform engineering because when you have hundreds of developers building and running applications on cloud-based, managed Kubernetes services, it doesn’t really matter how small the platform component is.
When you take into account that the work needs to be distributed across 10, 20, or 30 or more Dev teams (depending on the size of your organization), you realize the inefficiency of it all. It makes more sense to centralize that function and have one team responsible for platform engineering.
To learn more about new cloud native approaches for establishing security and observability for containers and Kubernetes, check out this O’Reilly eBook by Tigera.