Shippable ‘Formations’ is a Container CI/CD Without DevOps-Style Scripts
Last September, we introduced you to Shippable, which is a long-standing continuous integration and delivery services firm (dating way back to early 2013) that rebuilt its business model around Docker containers. Wednesday morning, the company announced the complete overhaul, once again, of its continuous integration/continuous delivery platform (CI/CD), this time with the objective of maintaining live, continuously updated container deployment environments for testers and for production, without the use of DevOps-style build scripts.
“The CI platform is getting a full upgrade,” said Avi Cavale, Shippable’s CEO, in an interview with The New Stack, “where containers are going to become first-class citizens. You can run Docker Compose, Docker Build, any of that kind of stuff on our CI platform, right from the get-go.”
Getting Rid of the Transients
Up until today, Shippable’s customers had been using orchestration tools such as Chef, Puppet, Salt and Ansible to manage containerized CI/CD environments hosted there. That requirement now goes away with the addition of what Shippable calls Formations.
“We are switching from transient containers to a persistent container system,” the CEO said.
Our Alex Williams likes to use the term “ephemeral” to describe one of the basic qualities of a Docker container: its temporary nature, its relative insignificance. It spins up, does its job, and then dies without a tremendous amount of drama.
At least that’s the plan. The way Docker had planned to make use of this ephemeral nature was by making containers transient — not just temporary, but returning resources back to the system when they exit. One method Docker devised for sharing persistent data between transient containers, as well as preserving that data once the containers do exit, was by producing volumes.
“Because volumes are isolated filesystems, they are often used to store state from computations between transient containers,” reads the current “Docker cheat sheet” on Github. “That is, you can have a stateless and transient container run from a recipe, blow it away, and then have a second instance of the transient container pick up from where the last one left off.”
In everyday practice, however, some people started questioning the necessity of what they considered going through the motions for Docker’s sake. With respect to continuous delivery, Cavale told us, the original architecture was causing way too much busy work just for the architecture’s sake.
Every time a transient container is pushed to GitHub or some similar repository, he explained, it generates countless workflows. Some of the processes within those workflows spin up new containers just for the sake of running Docker’s build command — containers which are themselves transient, of course, and are immediately killed off.
“We do about 200,000 containers a week right now,” said Shippable’s CEO, “in terms of these containers spinning up and down for our customer builds.” The end product of each of those containers is a push to Docker Hub or another registry. From there, Shippable’s customers go through the motions, once again, of instantiating those containers in their choice of environments. Shippable’s role was to automate a process that badly needed automation.
In the new system, all that “automation” is replaced with a new system of persistent containers — not transient ones whose purpose is to spin up new transient ones, but an environment that may more closely approximate, if not outright mirror, the production environment.
Reconciling the “Latest” Tags
Within this persistent environment, changes to designated container builds go live immediately across the board. Shippable’s hope here is to eliminate a myriad of problems with versioning and tagging, by treating the development system as one system rather than a space/time continuum that can exist in multiple time zones depending on where you happen to be sitting.
With containerized environments in typical CI/CD, there’s no clear standard around how different versions of containers are distinguished from one another using tags. Docker itself didn’t make this problem any easier, having implemented a practice where images pushed to a repository that omits any tag whatsoever, is automatically given the tag latest. Not that tags expire, or anything, or that a newly latest-tagged image overrides the existing latest-tagged image … or anything useful like that.
“It’s easy to understand why this may surprise people,” writes the consultancy ContainerSolutions.com. “Consider the sentence, ‘Just pull the latest image.’ Does this mean the image tagged latest, or the newest image in the repository? Are they the same thing? What does it even mean to be the newest image in a repo; is it the newest stable or the newest development image?”
Cavale calls the latest tag “one of the craziest things Docker supports … If anybody tries to deploy using latest, they’re automatically going to be in big trouble very, very quickly.”
What CI/CD customers are aiming for is an eventual immutable state for containers. Cavale has argued in the past that containers must be versioned throughout their evolution, in order that immutability remains a feasibility. Having argued that point publicly, he admitted, a sizable plurality of his customers disagreed. To which he responded, “If you start overwriting your version numbers, then you’ve just lost immutability because you no longer know what is running where.”
Whether a container truly is the same one across development, testing and production environments currently depends, Cavale points out, on when the docker run command was instantiated. “Now you’re in the same mess that you were before, which is, ‘I don’t know what is running where.’”
So Shippable’s hope is that equalizing these deployment environments, and trimming down the scripts necessary to deploy containers in those environments to essentially nothing, creates the possibility for developers to explore the virtues of immutability in ways that were not possible for them before.
Docker and Shippable are sponsors of The New Stack.
Feature image: “Looking straight into The Wave, Coyote Butte North, Arizona” by Frank Kovalcheck is licensed under CC BY 2.0.