OpenStack Spins Off Zuul, a Gated CI Pipeline for Multiple Repositories
Continuous integration (CI) software originally developed to manage the massive multi-contributor OpenStack software platform has been updated to handle other complex collaborative software projects as well.
Zuul offers an automated gated workflow pipeline, explained James Blair, principal software engineer with Red Hat and founding member of the Zuul project team, at the OpenStack Summit, being held this week in Vancouver. With gating, no new code moves to production until it passes a set of unit and integration tests defined by the keepers of the project.
With Zuul, “Humans don’t merge code, machines merge code,” further explained Monty Taylor, at the OpenDev, a collaborative event co-located with the OpenStack Summit. And it works with complex code sets: Zuul can coordinate integration testing across multiple repositories.
Monty Taylor, with the Zuul T-shirt
The OpenStack Foundation, which also manages OpenStack and Kata Containers, will steward Zuul going forward. This Zuul is not to be confused with the open source routing gateway from Netflix of the same name. It takes its name from a mythical demigod gatekeeper in the 1984 movie “Ghostbusters.” And like the fictional creature, this Zuul is also an unforgiving gatekeeper, not letting any code pass into production until it passes a series of tests.
OpenStack is updated, for now, once every six months. Making sure that all the components work together for each release is a formidable task, especially since they are spread out across nearly 2,000 repositories. Zuul makes sense of all this.
For OpenStack, a developer submits potential changes to Gerrit, which are then reviewed by project maintainers. If the maintainer approves the patch, Zuul will then run tests on it, and if those pass, it will be merged into the code base. “Every change that is proposed to a repository is tested before it is merged,” Taylor said. Although built to interface with Gerrit, Zuul can easily work with other software development platforms such as GitHub, and even work in an all-automated GitOps-styled pipeline.
In a nutshell, Zuul listens for things that happen with the code. When a proposed patch arrives, Zuul prepares the job config and procures the resources to complete the job. A companion service called NodePool handles the procurement of build resources. The Zuul executor uses Red Hat’s Ansible to run the job, captured on the Ansible playbooks. Zuul returns the results, and — potentially — merges changes (if they pass the tests).
With Zuul, each project teams writes their own tests, which can be shared with others. “Any Zuul system in the world can add the Zuul repository to their installation of Zuul and would have the benefit of having these jobs already pre-defined,” Blair said. OpenStack itself has created a number of tests that may be useful for other projects as well.
“Co-Gating”
With OpenStack’s multiple repositories, Zuul practices “co-gating” or the testing of software across multiple repositories. In OpenStack’s case, “It’s not enough that if a change to Nova works for Nova, if it breaks Cinder in a deployment of OpenStack, then its still not good,” Taylor explained. “OpenStack is the thing we are testing.”
Zuul was designed to work with multiple code changes through some techniques of parallelization. “The system will take care of the fact that 40 other people approved code for the project at the exact same time,” Taylor said. “It knows where to run things, and what order to run them in, what states it should run them in, and ultimately how to respond to the results.”
Taylor offered an example from OpenStack: Someone approves a change to the Nova component, so the system kicks in and starts running tests on the proposed patch. While that is happening, someone approves a change to Keystone, so Zuul starts running tests on that patch as well, running it in a full OpenStack deployment plus the first proposed exchange to Nova. If two more, separate, Nova patches are submitted shortly afterward, Zuul will test those with an OpenStack deployment, incorporating both the Nova and the Keystone patches, which are still being tested.
“We have four different sets of git states that represent speculative future states of the system,” Taylor said. “We are testing what will happen if we merge these patches.”
In this mocked up example, the Keystone patch fails, and those results are reported back to the user. As a result, Zuul cancels the tests for the two last Nova patches that came after the Keystone patch. Instead, they are automatically re-based on the first Nova change, and without the Keystone patch. When the first Nova successfully completes its test, the patch gets merged into the codebase. “This is now the new state of Nova,” he said.
“Just from the action of four developer approval events, we were able to spin through a couple of Git repository future states and automatically merge those changes through the process,” Taylor explained.
Zuul is not a general purpose automation framework, Taylor warned; it was not designed to replace Ansible Tower, for instance. It has been worked with virtual machines and with containers. The open source software has already been used by BMW, GoDaddy, OpenLab and Wikimedia, in addition to OpenStack itself. BMW, GitHub, GoDaddy, Huawei, Red Hat and SUSE have all contributed to the project.
The OpenStack Foundation and Red Hat are sponsors of The New Stack.
Feature image: James Blair, Red Hat.
