Yelp Releases an Initialization System for Docker Containers
Yelp, an early adopter of Linux containers, has released an initialization system that runs inside of Docker containers, called dumb-init, designed to eliminate signaling issues that can spawn zombie processes.
The company is in the process of moving individual commands into containers so they can slowly migrate tools and infrastructure without disrupting current workflow. This requires their processes to run as if they were running in the legacy system in regards to user input, responding to signals and dying when expected.
“Instead of forcing developers to unlearn their existing workflow, we can move individual commands into containers without developers even realizing they’re spawning a Docker container on each invocation,” explained Yelp software engineer Chris Kuehl, in a blog post announcing the release. “It also lets us take a practical approach to Docker in development: rather than require that everything live in a container, we can choose to use containers when it makes sense from a business or technical perspective.”
When running testing their apps within containers, the programmers encountered unexpected behavior, including orphaned zombie processes, relating to the signals being sent back and forth. Lightweight Docker containers still run a full Linux system, making the shell act as PID 1. It turns out this interrupts the signaling processes.
“Signals sent to the shell won’t be forwarded to the subprocess, and the shell won’t exit until your process does. The only way to kill your container is by sending it SIGKILL (or if your process happens to die),” Kuehl wrote.
It turns out when the init process terminates prematurely, then all children are terminated uncleanly by the kernel, leaving orphaned zombie processes. And nobody likes zombies in their processes.
Enter dumb-init, a simple solution to these problems. The code, added to the front of any container’s command, takes on the role of PID 1. This makes processes coming into the container as PID ~2, thus avoiding special kernel behavior specific to PID 1, and solving the signaling issues (e.g., orphaned zombie processes).
The One Thing Well technical blog summarizes dumb-init: “dumb-init is a simple process supervisor and init system designed to run as PID 1 inside minimal container environments (such as Docker). It is a deployed as a small, statically-linked binary written in C.”
While there are a few off-the-shelf products (runit or systemd), they run on Python, require some work to set up and more resources to run. dumb-init is, Kuehl said “a simpler way to do things properly: it spawns your process as its only child, and proxies signals to it. dumb-init won’t actually die until your process dies, allowing you to do proper cleanup.”
Simple, clean, easy to install. Sounds elegant. Best of all, you can check it out for yourself. Yelp gave dumb-init its own page on GetHub. Dumb-init is one of a number of internally-build tools that the social recommendation service has released as open source.
Docker is a sponsor of The New Stack.
Feature Image: “Tractor Stuck at Halfway Point” from the U.S. National Archives, via New Old Stock.