The Key to Successful Platform Engineering: Self-Service IaC
The cloud has always been technically self-service, but in practice it hasn’t been for many developers because of its complexity. It is becoming even more difficult to use as companies adopt modern architectures (cloud native, serverless, etc.) and new providers (multicloud, SaaS applications), and as cloud providers release more services.
That’s why competitive engineering teams are now figuring out how to multiply the output of their DevOps, site reliability and infrastructure engineers by eliminating bottlenecks to accelerate the productivity of all developers.
Platform engineering offers a solution by enabling development teams to build, deploy and manage their own cloud infrastructure on a self-service basis, while following best practices.
Let’s explore the benefits of platform engineering and the three main approaches that successful platform teams use to empower developers to self-service infrastructure, as well as the importance of choosing an Infrastructure as Code (IaC) framework that supports the approach used.
Developers need infrastructure to run their applications and services. Traditionally, many companies have had central infrastructure teams to provision and manage infrastructure on behalf of developers, but this model is prone to bottlenecks as developer requests for infrastructure overwhelm central teams.
As modern development teams have taken responsibility for owning and operating their own infrastructure, they also need simple and fast ways to provision it while adhering to best practices. You might think that central infrastructure teams are less needed with these shifts, but they have a critical role in making developers and the company successful in this new world.
While developers are increasingly responsible for managing infrastructure, it is impractical to expect all developers to acquire a deep understanding of cloud infrastructure.
There are several problems that platform engineering addresses from a developer-experience angle.
First, the cloud is too complex and unwieldy for most developers to use without abstractions and tooling. Modern cloud architectures, such as microservices and serverless, have increased the complexity of managing infrastructure. Many companies are running applications on Kubernetes. And while developers are increasingly responsible for managing the infrastructure that runs their applications, it is impractical to expect all developers to acquire a deep understanding of the intricacies of cloud infrastructure.
Second, developers need to know which infrastructure resources to provision and how to adhere to best practices determined by the company, such as security and cost concerns, cloud regions, regulatory requirements and approved cloud providers. With over 200 services to choose from on AWS alone, determining the right resources is not a simple task.
Third, developers need an easy way to provision, configure and manage the infrastructure. This can be complicated since many architectures use multiple resources from one cloud, they often include resources from other cloud or SaaS vendors, and there may be challenges when deploying something complex like Kubernetes.
By decentralizing management of infrastructure, companies have dropped significant new complexity onto the laps of developers. The need for abstracting that complexity away so that developers can focus on shipping features faster is more pressing.
That’s where platform engineering comes in. Platform teams build shared tools and services to help development teams develop, deploy and operate cloud infrastructure on a self-service basis. This includes cloud infrastructure, container orchestration platforms, databases, networking, monitoring, code repositories and deployment pipelines.
Effective platform teams treat the application developers that they serve as customers that must be understood and won over through products that solve their problems.
There are a few traits that characterize successful platform teams. These include a business shift from treating infrastructure teams as merely a cost center to an accelerant of the business. Typically, this means that the business has given the team the authority and resources to execute a platform engineering model. Effective platform teams also adopt a customer-driven mindset where they treat the application developers that they serve as customers who must be understood and won over through products that solve their problems.
Last, but not least, successful platform teams are staffed with talented engineers with the multidisciplinary skills, experience and empathy needed to build a great product, serve developers’ needs and “go to market” within their company. Often, they have experience with many engineering disciplines like infrastructure or DevOps and software engineering, and various job titles like software engineer, DevOps engineer, site reliability engineer, cloud architect, cloud engineer and more.
By providing developers with infrastructure and tools to deploy and operate their applications efficiently, platform engineers enable developers to focus on building great software.
Broadly speaking, there are three main approaches that platform teams are using to enable developers to self-service infrastructure. The first is to create infrastructure libraries that express infrastructure resources and configurations that are used to provision infrastructure using a deployment tool.
The second is to create infrastructure command-line interfaces, which allow developers to provision and manage infrastructure through a familiar CLI similar to Heroku. The third is to create infrastructure platform applications, which provide a graphical user interface for developers to provision and manage infrastructure.
Common across all these approaches are standardized architectures that bake in best practices defined by the platform team, mechanisms for enforcing deployment rules and uniform workflows for deployments (code reviews, pull requests, CI/CD, testing, etc.) The main differences are the interface that developers use to interact with infrastructure, and the approach selected typically depends on which works best for achieving the platform team’s goals.
To enable these approaches, platform teams need a deployment framework that supports their product. Although building your own custom deployment tool may be tempting, a much more efficient and functional approach is to adopt an Infrastructure as Code framework.
IaC allows engineers to define and manage infrastructure in the same way they manage code. This approach provides several benefits, including version control, testing, reusability and automation. Many frameworks exist, some are open source and others not, and they may be cloud-specific or multicloud. Most use YAML/JSON-based domain-specific languages (DSLs), while some use general-purpose programming languages.
While there are benefits and drawbacks to each approach, many platform teams are finding that adopting a general-purpose language for their IaC provides the most flexible, powerful and adaptable approach. These languages make it easier to model complex infrastructure because they natively support constructs like loops and conditionals, and they provide a familiar and mature model for abstraction and reuse.
Last but not least, it’s much easier for developers to use IaC when it’s written in languages they already know.
A great example of a company that has adopted platform engineering is Elkjøp Nordic, the leading consumer electronics retailer in the Nordics. The company had a modernization strategy to increase the agility of development teams by giving them ownership over their services and the infrastructure that runs them. At the same time, it wanted to create security and compliance guardrails that prevent issues while maintaining developers’ freedom.
It accomplished this by building an infrastructure platform application that enabled developers to provision infrastructure running on Kubernetes in Azure.
Using a web interface, developers can specify the infrastructure needed, such as type of environment and which members have access. The application provisions and configures all of the necessary infrastructure resources, including a GitHub repository set up for the project. The application was built using React, TypeScript and Pulumi as the IaC framework. Pulumi let the company define reusable infrastructure architectures in TypeScript. The application also used Pulumi’s Automation API to embed IaC capabilities within the application, such as provisioning, configuring and destroying infrastructure on Azure and GitHub.
The result was that development teams could independently provision infrastructure without incurring significant costs on the platform team and eliminating provisioning bottlenecks. Meanwhile, the platform team could ensure that developers were provisioning approved infrastructure, which reduces risk of misconfigurations. You can learn more about Elkjøp platform engineering in this blog post.
Platform engineering is a powerful and emerging engineering discipline that’s helping companies increase their developers’ productivity and gain more value from their cloud investments. By taking a customer-centric approach, platform engineering teams enable developers to focus on building great software by providing them with self-service infrastructure tools that meet them where they are and increase productivity. This tooling lets them provision the right infrastructure for their application and manage it.
Choosing an Infrastructure as Code framework is an important foundation of a platform engineering strategy because it defines how infrastructure is modeled, tested, distributed and deployed across an organization.
In particular, the choice of IaC language affects infrastructure’s usability, reusability and capacity for scale. General-purpose languages provide major advantages because they are widely adopted already, have more flexibility and expressiveness than DSLs, and come with rich ecosystems of tools and frameworks that increase productivity.
