TNS
VOXPOP
Where are you using WebAssembly?
Wasm promises to let developers build once and run anywhere. Are you using it yet?
At work, for production apps
0%
At work, but not for production apps
0%
I don’t use WebAssembly but expect to when the technology matures
0%
I have no plans to use WebAssembly
0%
No plans and I get mad whenever I see the buzzword
0%
CI/CD / DevOps

Imagine a Smarter CI Pipeline

What if CI pipelines had less boilerplate, faster runtimes, and could run from our laptops?
Dec 12th, 2023 9:07am by
Featued image for: Imagine a Smarter CI Pipeline
Featured image by Irene Dávila on Unsplash.

Imagine a CI pipeline.

It is a collection of jobs that can depend on one another. Each job has a series of steps where you can run commands and scripts (usually on an operating system other than the one you use when writing the pipeline).

Maybe you’re envisioning GitHub Actions, maybe CircleCI, or Jenkins. I tend to think about CircleCI.

As I’m writing this, the .circleci/config.yaml file in one of my projects is 1,251 lines.

It looks something like this:


I prefer these 1,251 lines to the GitHub Actions I use in another project. They are split over multiple files that are all in the same directory. Even though each file is short, I always get lost hopping between them — and I consider myself a pretty proficient Vim user.

Maybe there’s room for improvement here. How about we try to reimagine pipelines?

Imagine a Smarter Pipeline

CI pipelines are just script runners, and each job usually has a name like build_x,deploy_y and test_z. Oh, and run_that_script.

There’s a pattern here. Perhaps it would help if these jobs were a little smarter. Instead of being completely generic, you would have jobs of a specific kind. Like a build job, a deploy job or a test job. Oh, and a job for running that script.

I’m not just talking about a job with the name build_x, but rather a job of kind “build” where the fact that it’s a build job has some meaning.

It could be “smart” and save some lines of config. Instead of:


It could be something like:


It’s a nice thought.

But how much can a job of kind: build know? We could be compiling a Rust binary, bundling JavaScript or building a container image. There are still steps involved.

It would be nice to get a bit more specific. Say it’s a build job for some local package. Or a build job for a container.

Maybe something like:


Now we’re talking! A semantic CI pipeline that understands what we mean when we say “do a build of type container” and relieves us of writing all that boilerplate.

Heck, we could use this to describe all the parts of our system!

A Portable Pipeline

Except… it’s still CI. And it can only run from CI. That’s a bummer.

What’s that all about anyway? It’s kind of insane how much time we spend writing configs that we can test only on someone else’s “computer”— by opening a pull request!

I mean, think about it.

It’s like asking a web developer to use their editor, then send the code to a friend who owns a browser so the friend can report back whether the font size looks good or not.

Wouldn’t it be nice if we could run these pipelines from anywhere?

What is a pipeline anyway? Isn’t it basically a blueprint of all the things that need to happen to build, test and deploy your system? Sounds pretty useful. Would be neat to reuse that blueprint for other stuff as well. For example, development.

Imagine being able to run your pipelines from your terminal as you write them. Or even better, to run specific parts of your pipeline. For example, only the build jobs or maybe a specific test job.

Since we’re imagining things, how about we also reimagine those 1,251 lines of CircleCI config or that wad of GitHub Actions? We’re building distributed systems, so why not distribute the config? Remember those container-build jobs we talked about? How about we just place them next to the thing we want to build? We co-locate other builds, deploys and tests next to the things they’re meant to build, deploy and test.

Surely there’s a smart way to scan for all these configs without having to stuff them all in the same file or same directory. We could even scan across repos. Code repositories are just a means of organizing our code. Some teams like monorepos; other teams use multiple repos. It shouldn’t really matter if there’s a smart pipeline tool that can figure things out.

In fact, at this point, it’s not much of a pipeline anymore. It’s an executable blueprint. It’s the fully portable architecture of your system as code.

And a Faster Pipeline

At the start, I mentioned that we often write pipelines on one operating system and run them on another. That doesn’t agree well with portability.

So I guess our smart tool needs to run these jobs independently of the host (such as a laptop or CI runner). How about in the cloud, since we are cloud native?

But wait… there’s another opportunity here.

If we use this imagined tool to run, for example, a test job from a laptop “in the cloud” and then the CI runner uses the tool to run that same test “in the cloud,” isn’t it basically doing the same thing twice (assuming the environments are the same)? Do we need to waste all those CPU cycles?

Therefore, the final ask of this smart tool is that it keeps track of all the files and configurations that belong to a given job — and all its upstream dependencies — and never runs the same job twice. So even if our CI pipeline contains hundreds of jobs, maybe only a couple need to run.

That’s no small feat, but hey, we’re just letting our imagination run wild. Dreaming of that dev tool we wish we could build.

You Can Stop Imagining

You’ve probably guessed it by now, but that’s what we did build. It’s called Garden, and we call them actions, not jobs. Garden allows you to create a blueprint of a software system of any complexity across multiple repos with just simple primitives.

Its plugin system allows you to build containers, apply manifests, install Helm charts and plan Terraform stacks without having to spoonfeed each step into a system that’s not smart enough to be of any real use.

There’s actually a lot more to it than this. You can get started right away, or check out this video to see it in action.

Group Created with Sketch.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.