Modal Title
CI/CD / Containers

Chef Working on a Build Service to Automate Habitat Packaging

Jul 15th, 2016 9:50am by
Featued image for: Chef Working on a Build Service to Automate Habitat Packaging

Quickly extending its recently launched and much-discussed Habitat application packaging tool, Chef is working on a component, called Builder, that will provide an automated build service and package repository for the open source software.

Although the company has not released this functionality yet, these capabilities, once finished, could put Habitat on par with other automated container packaging systems, such as Docker Build. Chef provided a glimpse into the technology and architecture at its annual ChefConf user conference, held this week in Austin Texas.

“Builder is a software-as-a-service platform to build your Habitats,” said Jamie Winsor, a Chef programmer who has been working on Builder since October.

The software will also capture the history of builds. “Having the build results next to the package is really useful for the user,” Winsor said, noting today developers usually have to hunt down the build results of a package. “Anyone who runs an apt-get install doesn’t know what they will get.”

Chef may use Builder to offer a hosted public build service, though organizations will also be able to use the software to set up an in-house build service and package repository.

“No one has ever built a build service where you can bring your own build node,” said Chef chief technology officer Adam Jacob. Docker Hub, for instance, requires users to deploy Docker’s build mechanisms. “For your account, you may want to decide to build it on-prem for your infrastructure, but that doesn’t necessarily mean you’d want to run all the infrastructure to trigger your builds,” he said.

The idea with automated build systems is that they can be hooked into larger continuous integration/continuous deployment workflows, providing a smooth pathway from development to deployment.

Builder could be used to monitor a shop’s codebases on Github, and, when a developer uploads an updated package, it will automatically kick off a new build process and publishes them to a Habitat Depot (Chef’s equivalent to a Docker Hub).

The company is currently in talks with Microsoft, Intel, Google, Amazon to look for ways to insert Habitat in the larger ecosystems, Jacob said. The company is also mulling the possibility of having Habitat’s Builder serve as the build pipeline for the company’s newly launched Chef Automate, connecting Builder with the platform through a gateway with Chef Delivery, Automate’s workflow component.

Habitat itself, also a component of Automate is a program that streamlines the process of packaging software so it can be easily deployed in any runtime. Just Chef’s namesake software automates the deployment of IT infrastructure, Habitat offers a similar solution for applications.

Using the program, a developer can download an application package from a Depot, tweak the environmental variables and set configuration settings so that it can be deployed automatically. This could be especially valuable for distributed applications, such as the Redis database, that have multiple components that need to find each other and self-organizing before going live.

One of the advantages that Habitat could bring is the ability to package an application so it can be run within multiple environments, not only containers but also for virtual machines, Amazon Application Machine Images.  “The automation travels with the application,” Chef people were fond of telling attendees at the conference.


In one conference session, Winsor delved into the architecture of Builder, with the idea that partners would augment the technology and adapt it to their systems.

Written in Rust, Builder is a service-oriented design, fronted by an HTTP gateway that communicates with a set of back-end microservices. Requests can be made by a number of ways. One is from a Web client, which was written in Angular 2. There is also an HTTP client that can ferry application JSON to a REST endpoint.

The messages are sent to the HTTP gateway, which provides a RESTful interface into the cluster. It also handles user authentication, through OAuth. The gateway nodes transform the REST messages into messages formatted with the ZeroMQ socket library and Google Protobuf message protocol.

When a job server receives the request, it forwards the request to a handler, which carries out the specific task. The job server also copies the job information into a database, creating a serialized record accessible by future sessions.

The handlers are OS-specific and work through the queue for their OS, sending a message whenever they are free for another job. The builds themselves are carried out within Habitat Studio, which creates the packages within an isolated environment, and then publishes the package to the public depot.

Notice of a job completed is sent back to a router node which relays it back to the gateway.

Chef hopes that, over time, contributors will build their handlers. The current handlers are single-threaded, making them easy to understand for a coder with little system programming experience. Creating a new one would involve writing a function and linking it to a unique message ID so it can be called.

Distributed and Self-Updating

Builder is a distributed system and is broken into multiple shards or instances. There are multiple router nodes and session nodes in each shard. There may be multiple databases as well.

This approach helps localizes failure. “When a pull service goes down, we have a partial outage for all users. If a session goes down, you can’t make any more authenticated requests. If a job server goes down, you can’t make any more jobs, but you can still log in and get your packages. If the depot goes down, you can’t download packages or put them in but you can still create jobs,” Winsor said.

If a shard goes down, then only a subset users will suffer a partial outage.

One of the most interesting aspects to Builder is that it is self-hosting. “Habitat supervisors all the Builders. Habitat supervises all the processes,” Winsor said. “The supervisors watch the public depot. Habitat is running builder and builder is looking at itself, which is running. When Builder builds itself, it sends the package to itself, and it notices it got updated, and then it all updates.”

Docker is a sponsor of The New Stack

Feature Image: Austin street art, painted by Lucas Aoki Córdoba.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack, Docker.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.