Flatcar Linux: The CoreOS Operating System Lives on Beyond Red Hat
As part of the integration between the product lines of the company that put containerized Linux on the map and the most powerful name in all of enterprise Linux, Red Hat has said it will adopt the “CoreOS” name for its forthcoming edition of its own Linux for containers. Yet although “Red Hat CoreOS” will adopt the self-updating technology that made CoreOS Inc.’s products competitive, a Red Hat official told The New Stack, the kernel will remain essentially compatible with the Fedora/Atomic/RHEL base it has already produced, for reasons the official said mainly concerned maintaining the certification of systems running on that kernel.
So while Red Hat makes plans for what to do with the CoreOS Container Linux, others are also eyeing the code base which after all, continues to be an open source project licensed under Apache 2.0. During the last KubeCon + CloudNativeCon in Copenhagen, attendees were re-introduced to Kinvolk, a Berlin-based group of open source contributors, including Chris Kühl, who were early contributors to the rkt container runtime devised at CoreOS and since donated to the CNCF. Now, Kühl and his colleagues have committed to producing and maintaining a fork of CoreOS Container Linux. Called Flatcar Linux, its immediate goal is to maintain its container-agnostic architecture, and maybe later try resuming its own development path.
“With Container Linux, CoreOS created an OS that is pretty close to ideal for cloud-native infrastructure,” stated Kühl in a note to The New Stack. “When the acquisition was announced, there was a lot of confusion about what would happen to it. Thus with Flatcar Linux, we first wanted to ease those concerns by offering a drop-in replacement.
“But Flatcar Linux was likely to happen even before the acquisition was announced,” he continued. “We had been getting requests to support Container Linux and, as we mention in the FAQ, we didn’t see any means of providing commercial support for an OS without controlling the full build pipeline and maintaining it.”
Rebooting the Bootstrap
On April 30, the Kinvolk group officially released Flatcar Linux as a public project, with its own repository. A check of the first draft of its documentation reveals the group is obviously continuing CoreOS’ work in making container configuration files easier for human beings to produce. (Compare, if you will, CoreOS’ draft, while it still exists, to Kinvolk’s.) The predecessor company’s Container Linux Config Transpiler project, still on GitHub, enables a user to draft a more readily legible YAML configuration file, in place of a more explicit and, presumably, difficult-to-assemble JSON file. The Transpiler converts the former to the latter, adding a step to the configuration process but reducing much of the headache involved.
Red Hat’s Atomic had been using the cloud-config and cloud-init syntax that the original CoreOS was steering away from. It’s a safe bet that the new “Red Hat CoreOS” will continue along Atomic’s path for this aspect of the system.
“We are big supporters of Kubernetes,but we think Flatcar Linux should stay a minimal distribution, and do not want to tie it directly to a specific container orchestrator.” — Chris Kühl
“As far as Red Hat CoreOS goes, the changes that Red Hat is proposing to make a lot of sense for Red Hat and are technically sound,” remarked Kühl. “But these changes could easily be seen as a case of acquisition-driven engineering, which is often necessary for the acquiring company, but add little value to users. One would be hard-pressed to find a Container Linux user who was asking for the changes proposed for Red Hat CoreOS.”
One example Kühl cited was what he claims to be a proposal that Red Hat, going forward, include Kubernetes’ kubelet — the remote agent that reports on the health of its node within a cluster — with future builds of its containerized OS. “We are big supporters of Kubernetes,” Kühl told us, “but we think Flatcar Linux should stay a minimal distribution, and do not want to tie it directly to a specific container orchestrator.”
Among some in the Linux community, there is a perception that the lifecycle of Red Hat Enterprise Linux, and the kernels in RHEL’s orbit, has become untenably long. In a Hacker News thread recently launched by OpenShift senior product manager Jimmy Zelinskie, a loyal CoreOS customer remarked, “One of the main reasons we use CoreOS is that it ships stable and state of the art features from the latest Linux kernels, so features and infrastructure like eBPF are available and can be used without problems. RHEL kernels… on the other hand are ancient and heavily outdated, and only backport some of the newer features from upstream kernels, so user space making use of it cannot be run anymore preventing latest innovation [sic].”
That provoked Zelinskie to remark, “RH CoreOS is not going to have a kernel that is synchronized with the RHEL release cycle. We’re working to find out what types of requirements for certifications that we want to support and, using that as a guideline, we’ll be shipping the freshest possible kernel.”
But although that kernel may be called “CoreOS,” it will not be the same kernel that Flatcar’s config system, at this moment, refers to in its native syntax as coreos.
There’s also the matter of including OSTree and Ignition, two components that derived from Atomic and Container Linux, respectively. OSTree (represented in Project Atomic as rpm-ostree) is a means of packaging, deploying, and managing immutable file system trees. This way, when downloading the latest components — for instance, “over the air” — a repository can deliver the unaltered source modules from all the necessary components, and the deployment system can perform the merges on the client side, resulting in a clean and up-to-date deployment
Ignition, on the other hand, is a system CoreOS Inc. devised to pre-provision the resources a Container Linux system would need, prior to its being bootstrapped. It enables a user to draft a more readily legible JSON configuration file, in place of a more explicit and, presumably, difficult-to-assemble YAML file. As such, it would conceivably substitute for the cloud-init system that the Kinvolk team is continuing work on.
In his Hacker News thread, Red Hat’s Zelinskie explicitly stated that future editions of OpenShift would support both Container Linux and Red Hat CoreOS. In so doing, OpenShift would include “a Linux distribution that leverages Ignition and immutability to provide the minimal environment needed to run Kubernetes/containers.”
Yet the Flatcar Linux project’s Kühl told us, “We also don’t see a pressing need to add OSTree, which seems to be a part of the plan.” Without mentioning Ignition specifically, however, Kühl made a statement that implied its removal from Flatcar would not, in his view, be an improvement.
“In our view Container Linux is a mature product that needs little, if any, changes,” Kühl told The New Stack. “We are committed to remaining compatible with the upstream project and maintaining it as a solid platform for container workloads. Any changes we think are necessary will be proposed to the upstream project and then merged into Flatcar Linux.”
Re-rigging the Kernel
There will, however, be “custom, purpose-built Flatcar Linux builds” going forward — a means, Kühl said, for Kinvolk’s engineers to conduct experiments on the OS for special use cases. Kinvolk managed to hint at one such use case in a company blog post last April 25 with the curious title, “Towards unprivileged container builds.” In it, Kinvolk technical lead Alban Crequy touts the benefits of a system for packaging and reconstructing file systems from an archive — particularly for use by a container — that would not require root privileges.
It’s a concept we’ve discussed here in The New Stack a few times over the past two years. In the summer of 2016, it was a movement sparked by software engineer Jessie Frazelle — then with Docker Inc., now with Microsoft. Without the sudo requirement, Frazelle proposed at the time, an automated system could build a string of containers with much fewer steps, though with security still provided — just not bolted on. In recent days, Frazelle’s work has flourished at Microsoft, becoming the “Hard Multi-Tenancy” project for Kubernetes — just the latest example of a lucrative idea sparked by Docker, but continued elsewhere.
Crequy’s post explained probably all the reasons why the Docker architecture, with its build command, requires root privileges. It’s generating a file system and namespace based on the basic principles of Linux architecture — principles that may not necessarily have to stay valid if we can finally assume that CoreOS, and others, evolved the Linux kernel from that basic state.
When asked about whether the Crequy post provided a hint at Flatcar Linux’ future, Chris Kühl relented. “One can see that there are a number of kernel patches needed to enable unprivileged container builds as described in the post,” he told The New Stack.
“It’s fair to say it will take some time for those to get into the mainline kernel,” he continued. “To get this functionality to users more quickly, one of the purpose-specific builds we are looking to create is an image that will have the needed kernel patches to enable unprivileged container builds. We’re actively looking to cooperate with others who are interested in making this happen.”