A Security Comparison of Docker, CRI-O and containerd
The depreciation of dockershim in Kubernetes 1.20 received a lot of media attention. Kubernetes has reached major mainstream adoption, so any major change impacts a lot of applications. Docker was the original container runtime for Kubernetes and remains a very popular container development toolset and container runtime.
Many managed Kubernetes offerings have switched to other container runtimes. For example, OpenShift switched to CRI-O for its default runtime in OpenShift 4 in June 2019 and Azure Kubernetes Service set its default runtime to containerd in January 2020. Others have covered how images built by Docker will still work with other Open Container Initiative (OCI) compliant runtimes, and Mirantis has announced they will continue to support dochershim. However, most users will likely turn to the default runtime for their Kubernetes environment and will not notice the switchover.
In fact, for the average user, switching to new runtimes will produce significant benefits.
A Brief History of How We Got to Docker Deprecation
Docker used to be a monolithic tool that contained the ability to set up containers along with a myriad of developer tools. It contained a CLI, logging, storage management, networking, build tools and many other features outside of the core capacity to create a container. In the vein of Unix philosophy, however, Docker eventually broke up these components and contributed containerd, the container runtime component, to the CNCF.
The default configuration in Kubernetes kept Docker as the abstraction layer on top of containerd, which in turn was an abstraction layer on top of runc. This additional layer — and all of the additional tools that Docker includes — creates maintenance headaches, significant overhead and a larger attack surface for exploits. This is not ideal in ephemeral environments where teams are deploying code multiple times a day. The additional layer also introduces serious security implications, which we’ll discuss later.
At this same time, other container runtimes began popping up. Kubernetes maintainers came up with a standard called the Container Runtime Interface (CRI) that would be a common language for communication between kubelets and container runtimes. As Docker wasn’t compliant with this standard, it required a middle layer, or shim, to operate.
Kubernetes and containerd maintainers added a CRI shim for kubelets to talk directly to containerd. This allowed Kubernetes to cut out Docker and use containerd directly. This technically reduced the container capabilities, but those capabilities were unnecessary to begin with. For example, containers managed by Kubernetes don’t need SSH access. However, the CRI shim still added complexity and another attack surface, so developers eventually added the CRI natively into containerd as a plugin.
In 2016, CRI-O was developed as an alternative to Docker. It jumped ahead of containerd’s evolution to include a native CRI from the beginning. In this way, the kubelet talks directly to CRI-O via the CRI to pull an image and launch the lower-level runtime (e.g., runc), which in turn sets up the namespaces, cgroups, root file system, storage, several Linux security modules and conmon, a CRI-O specific monitoring tool. One important difference between CRI-O and containerd was the removal of some Linux capabilities, which we’ll cover in the next section.
Different container runtimes
What Do These Differences Mean for Security?
At their root, all three runtimes — Docker, CRI-O and containerd — pull an image, then spin up a lower-level runtime to configure and launch the container’s components and processes. Most of the attack surfaces are similar:
- Pulling malicious or outdated images
- Leaving hardcoded secrets in images
- Granting excessive privileges to a container, e.g. sharing host namespaces, host networks or applying the privileged flag
- Becoming a “noisy neighbor” by spiking CPU, RAM, Network, IOPs or disk usage to disrupt neighboring containers
- Exploiting kernel vulnerabilities
However, based on the architectural differences in the runtimes, each has some unique attack vectors. Docker is the most bloated of the three. For example, it contains a CLI and SSH daemon, opening up more ways for attackers to gain access to a container.
Containerd removes many of these features and reduces the codebase significantly. Because operations teams interact through Kubernetes’ control plane, these features aren’t missed. However, containerd still has a few unnecessary Linux capabilities, such as audit_write, mknod, net_raw and sys_chroot. These were defined by Docker maintainers early in its development before anyone knew exactly what features it would need, but these definitions were done at the expense of security. Despite its decreased attack surface, containerd was vulnerable to several attacks over the years such as poisoning images pulled from registries and container escape for host network containers, among other attack vectors.
CRI-O, in comparison, removes those Linux capabilities to reduce the attack surface. Despite this extra protection, CRI-O is not without its faults. For example, conmon is a useful monitoring tool, but has also been the cause of a container escape vulnerability. And previous versions didn’t use TLS with registries, opening up an opportunity for man-in-the-middle attacks.
What Can I Do to Protect My Systems?
For most Kubernetes users, the best advice is to move to the lower level runtimes, such as containerd and CRI-O. Their smaller attack surface will be easier to secure. Regardless of which one you choose, make sure to:
- Update as frequently as possible. If you are using a managed Kubernetes provider, upgrade to its most recent version.
- Use the least privilege model for your containers–avoid running containers as root, and strip away unnecessary Linux capabilities.
- Ensure images are updated, encrypted, signed and pulled from a trusted registry.
- Don’t assume an image is safe because it is open source.
- Ensure secrets are encrypted at rest and injected safely.
- Protect your hosts from exploits such as container breakouts.
How Security Tools Can Help
Cloud native security tools can create guardrails that help our teams follow the best practices outlined above. Security compliance tools can check that all nodes’ operating systems are up to date and that packages (such as container runtimes) are updated and patched. They can also help spot configurations in Kubernetes manifests that create containers running as root and either block or alert on those issues.
Some comprehensive platforms can integrate with CI/CD tools to scan images for vulnerable packages as a part of the build process (in registries or at runtime) and then identify which packages have patches available. Newer machine learning tools can build models of container traffic and alert on or block patterns that deviate from normal to prevent bad actors from performing attacks that exploit zero-day vulnerabilities.
Additionally, secrets stores such as Vault manage the distribution of secrets to enforce trusted access by injecting variables only when needed by a container.
Container runtime security
Moving to CRI-O and Containerd to Improve Security
The deprecation of Docker is less scary than it appears at first. Many organizations will benefit from the increased performance and decreased attack surface. Removing unnecessary bloat and adding native CRIs locks down containerd and CRI-O. However, all three are still exposed to vulnerabilities and misconfigurations.
For additional security measures, image scanning and runtime protection can secure Kubernetes-based applications. During the transition, make sure your security tool secures your runtimes and protects against these threats at every stage in the software lifecycle.