What Infrastructure as Code Must Do for CI/CD
The use of Infrastructure as Code (IaC) to provision and deploy infrastructure consistently and efficiently across various environments through a command line lends itself well to CI/CD. By applying IaC across the production pipeline, organizations report gains in productivity and resource savings.
However, not all IaC solutions are created equally. Prospective IaC users should scrutinize if and how a particular solution can really improve the CI/CD process.
An IaC solution should provide CI/CD with:
- Automated provisioning.
- Immutable version control, so that a single repository is used to create and remove infrastructure.
- Testing capabilities throughout CI/CD.
- The ability to set policy.
- The ability to manage security.
The Immutability that IaC offers CI/CD “is king,” Torsten Volk, an analyst at Enterprise Management Associates (EMA), told The New Stack. It earns this status, he said, because it ensures consistency, a clear audit trail for easy rollbacks, unified control over security and compliance, and overall efficiency,
What a proper IaC solution should offer CI/CD and why were key themes discussed during the PulumiUP virtual conference, Pulumi’s annual user’s conference on June 15.
Flexible Choices of Programming Language
Pulumi’s approach to IaC appears to be contributing to rapid growth in its adoption. While its rival Terraform claims nearly 10 times the market share of Pulumi, Pulumi’s share is growing at about two to three times the rate of Terraform’s, according to EMA.
Interesting how #Pulumi gives you the choice of programming language for #IaC. @PulumiCorp CEO @funcOfJoe noted during his #keynote you can ‘bring your favorite’ including #JavaScript, #typescript #Python, etc. #PulumiUP #sponsored @thenewstack #IaC https://t.co/XqtHSdAzeZ pic.twitter.com/z6KMsr414f
— BC Gain (@bcamerongain) June 15, 2023
The flexibility Pulumi offers a choice of programming languages is key. For instance, today many users are unfortunately limited to using YAML only for their deployments, which should not be the case; developers shouldn’t be limited to one language.
Pulumi, on the other hand, claims to support all major programming languages, thus offering more freedom of choice and a more direct way to provision infrastructure for CI/CD and in general.
“Pulumi is IaC in your favorite language — folks familiar with IaC may have experience with other tools that use domain-specific languages or even markup languages like YAML or JSON and oftentimes that works fine for getting started,” said Joe Duffy, Pulumi’s CEO and founder, during his keynote at PulumiUP. “But especially as we scale to modern cloud architectures, the cracks begin to show.
“So, Pulumi’s approach really is to take a different approach: Bring your favorite programming language, whether that’s JavaScript or TypeScript, Python, Go. Pulumi is multilanguage at its core. And what that means is you get all of the richness of programming languages to express your IaC.”
In Pulumi’s case, Duffy said, the adoption of IaC doesn’t mean your organization must forgo the use of GitHub or GitLab for pull requests and such that are integrated with CI/CD.
“If you’re already doing CI/CD with GitHub actions for GitLab pipelines, you can just use that to change from just application delivery to infrastructure delivery as well. And then of course, integrating with things like Docker and identity providers such as Okta,” Duffy said. “The net result of this is that the cloud goes from being drudgery to productivity — you just get a lot more done in less time, and it’s a lot more fun, too.”
Improvements in productivity can be“really an order of magnitude,” Duffy said, adding that Pulumi can be used to deploy an Elastic Kubernetes Service cluster on Amazon Web Services in under 15 lines.
“We’re not talking about 10%,” he said. “You’re going to feel like you’re coding much faster and getting a lot more done with a tighter inner development loop.”
@PulumiCorp CEO @funcOfJoe on enforcing standards: When it’s just #IaC for 1-3 practitioners, “it’s a very different thing than #IaC across your entire team..you start to think about how do we enforce standards?” #PulumiUP #sponsored #keynote @thenewstack pic.twitter.com/UmMulXLtcF
— BC Gain (@bcamerongain) June 15, 2023
IaC for Scaling up
Moreover, Pulumi aims to enable companies to scale their operations. While it’s easy to get started with Pulumi, the idea is to be able to support CI/CD teams deploying across different environments.
In this context, IaC should encompass tasks such as ensuring compliance, standardization and security across multicloud or diverse environments. It should act as a unified interface, facilitating management without the need for multiple tools or interfaces.
“Maybe you’ve got one development environment and next you’re going to think about going into production,” Duffy said. “And that’s where you start to think about standardization. When it’s just infrastructure as code for one, two or three practitioners. It’s a very different thing than adopting infrastructure code across your entire team.”
@starburstdata‘s Matt Stephenson showed how his org eliminated many headaches associated with CI/CD with #Pulumi and Automation API. #PulumiUP #sponsored @thenewstack https://t.co/XqtHSdAzeZ pic.twitter.com/C0j42rtIX2
— BC Gain (@bcamerongain) June 15, 2023
At Starburst Data, Pulumi is mainly used to perform complex CI/CD workflows, Matt Stephenson, a senior principal software engineer at Starburst, said during a conference talk.
Starburst’s infrastructure required to support its deployments is complex and vast, covering 20 different cloud regions through different cloud providers. Pulumi has helped to support very complex rollback processes, within the CI/CD workflow, Stephenson said.
“We want our infrastructure to be ubiquitous and approachable,” he said. “We want all engineers to be able to be comfortable going and making the changes they need to provision the infrastructure they need to for their product changes.”
The AI Component
More recently, AI has been showing how it could play a key role when using IaC for CI/CD in the very near future.
“AI integration lets developers pick their preferred development language and allows them to define their target infrastructure environment in plain English, to then use [Chat]GPT to automatically generate the code required to stand up the environment,” Volk told The New Stack.
With Pulumi Insights, Pulumi makes heavy use of ChatGPT functionality. During a presentation that Aaron Kao, Pulumi’s vice president for marketing, gave before the conference, Kao demonstrated to this writer how commands given to Pulumi Insights in basic conversational English allow the user to make queries and get actionable information about infrastructure across complex environments, such as Kubernetes clusters deployed across multicloud environments.
Kao showed how it is possible to input requests in simple English such as, “’I want an S3 bucket and an API gateway and a Lambda function. I want to build a dynamic serverless website. But if you don’t know, just give me a static website behind the CDN.”
The result is that “it’s literally going to spit out a program that gets you 80-90% there,” Kao said.
@PulumiCorp CTO @lukehoban: “How do we apply developer tools to #AI? How we think about programming these things and building systems on top of these technologies and what that means for developers–this is a very different kind of thing.” #PulumiUP #sponsored @thenewstack #IaC pic.twitter.com/aetneZg36E
— BC Gain (@bcamerongain) June 15, 2023
Without a doubt, AI will have a strong impact on CI/CD and IaC in the not-so-distant future. Today, therefore, IaC tools for developers should look and feel a lot different.
“These systems are different kinds of software to build than what we’ve been used to in the past and then the way that we approach building and testing is very different,” said Luke Hoban, Pulumi’s CTO, said during the conference panel: “AI and the Future of Cloud Development.”
“I’ve been very interested to see how that’s evolving and how we’re all learning a new set of skills for how we build different kinds of software.”
