Self-Service Infrastructure as Code in a Dev Portal with GitOps
Not so long ago, GitOps took DevOps by storm, offering a smoother and faster software delivery experience. There are numerous advantages to GitOps, making lives simpler for developers, DevOps and machines (workflows and pipelines).
However, it isn’t all smooth sailing, since GitOps requires that developers be familiar with a lot of DevOps practices and know-how. For example, Infrastructure as Code. IaC is an entire domain of expertise. Working with it requires compliance with applicable DevOps team standards and syntax.
For instance, suppose a developer needs to set up MongoDB for the microservice they are working on. It can be daunting for some developers to independently create the IaC and submit a pull request for GitOps to handle the application of the Terraform file. It also requires a considerable degree of trust. However, this is difficult to do on a grand scale, since not all developers have the required DevOps expertise.
Letting developers roam free with IaC raises other issues. Writing Terraform code requires an understanding of security best practices, and inexperienced developers may unknowingly introduce security vulnerabilities into the infrastructure.
This can lead to data breaches, data loss or other security incidents. Additionally, inconsistent code quality might also be an issue, as developers may have different coding styles and standards, making it difficult to maintain and update the infrastructure in the future.
All this is a developer experience problem, and the new way to solve it is to use internal developer portals (IDPs), the core developer-facing tool in platform engineering. Developer portals offer pre-made golden paths through a product-like user interface, which allows developers to perform many self-service actions from provisioning testing environments to rolling back a deployment.
Let’s explore how developers can perform IaC self-service actions, with GitOps under the hood. In this case, IaC file creation is handled automatically by the existing GitOps workflow for applying IaC files.
Step 1: Identify Self-Service Operations (Implemented as IaC Files)
First, identify the operations you would like developers to self-serve. Examples are:
- Create an S3 bucket/MongoDB
- Provision a developer environment
- Create an AWS account
Here’s an example Terraform file you might already have, which you want to provide as a self-service action through the internal developer portal.
Step 2: Create the Experience for Developers to Consume IaC via UI and API
Now we want to create the form you want developers to use when they use the self-service actions. Forms (and wizards) are about reducing cognitive load and providing a product-like experience. You can also make this easier to consume using an API. In either case, this defines the golden path, showing all IaC elements that matter to developers and keeping the rest under the hood. This solves the separation problem that often occurs with developers accessing GitOps, where some variables are for DevOps and some for developers, creating room for error and slowing developers down. When creating the UI form, think of the simplest experience you would like to provide developers. Add tooltips where needed, so that no questions remain unanswered.
Here’s an “Add DocDB” example:
Step 3: Auto-Generate IaC with the Injected Values from the Developer Self-Service Form
This is where we connect the dots.
We take user inputs from the self-service form and translate them into IaC parameters. Once the form is submitted, this will automatically generate an IaC file.
Step 4: Commit and Open a Pull Request for the Generated IaC File
To do this, we will implement a listener for form submissions that will create a pull request to the selected file. In most cases, DevOps engineers will be those that approve the action.
Here’s the pull request for this example.
Step 5: Update the Internal Developer Portal’s Software Catalog
Internal developer portals also contain a software catalog that shows much more than microservices. It contains what matters: CI/CD flows, clusters, dev environments, pipelines, deployments and anything cloud. Once a self-service action happens, the software catalog is kept up to date with the process of the triggered GitOps workflow (either an Azure Pipeline, Github Workflow or any other implementation to handle the IaC apply).
Here you can see how I used Port’s GitHub workflow provider to keep Port’s software catalog updated with the newly requested infrastructure.
You can see the entire flow in action here, both from the developer point of view and the platform point of view.
That’s it! You have managed to implement an end-to-end flow that lets developers add IaC to their applications with a click of a button experience, using existing GitOps implementations.
Underlying Developer Portal, IaC and GitOps Architecture
Let’s take a look at the architecture and how the developer portal interacts with GitOps and then updates the software catalog.
- Users perform a self-service action in the developer portal.
- Actions are then stored in a queue with Kafka.
- A centralized handler listens to the form submissions. In this case, it’s Port’s GitHub app, which both listens to the form submissions and handles the Terraform file generation.
- Once the Terraform file is ready and contains the relevant parameters, it will be committed and a pull request will be created.
- Once the PR is merged, the GitOps workflow, which was already implemented in advance, triggers an Azure Pipeline (or any other CI) that handles Terraform
apply. - As part of the Azure Pipeline, the software catalog data is kept up to date with the progress of the specific IaC action as well as adding/modifying/deleting a resource from the catalog according to the Terraform file
apply/destroyaction.
IaC Self-Service from the Developer Point of View
Let’s see what the developer experience is when using self-service IaC actions in the internal developer portal.
This is the form filled by the developer-user:
Since the IaC action may take time, it is best to show the developer how the action is progressing, as in this example:
Once the operation is complete, the developer will see the result of the IaC action in the software catalog that exists in the internal developer portal:
IaC Self-Service from the Platform Point of View
This is the audit log. Here you can see the status of different self-service actions triggered by the developers, including their status, the initial entity they were invoked on and the duration in case they finished running.
Here we can see the metadata for a specific invocation of a self-service action. Including the user inputs, general metadata of the action and the list of affected catalog entities (either new, updated or deleted as a result of the action).
As part of the metadata of the invoked action, it is also possible to attach a list of relevant links. In the IaC use case, a good link to attach would be the link to the pull request that the self-service action created with the new terraform definitions and their expected effect on the infrastructure.
As next steps, you might want to allow deletion of a database or perform a Day 2 operation, such as increasing resources via self-service actions. And of course, the sky’s the limit with the many actions you can allow your developers to perform.
Conclusion
Platform engineering is about creating reusable elements, and IaC actions should be part of that. To avoid issues with cognitive load and onboarding developers to work with GitOps, internal developer portals offer a broad self-service capability for developers, who can then easily perform IaC actions and have the software catalog update accordingly.
When the developer portal is decoupled from the underlying implementation, developers get a consistent experience while DevOps can change the way the logic is implemented under the hood. It’s also important to have the self-service pillar of the developer portal “speak” natively with the software catalog and not manage the self-service actions and the software catalog as two separate elements.
