Docker’s Unikernel Purchase and the Changing Role of the OS
Thursday’s announcement that Docker Inc. is purchasing Cambridge, U.K.-based Unikernel Systems brings to the forefront the question of the role of operating systems in the modern data center stack. It’s a question that open source developers firmly entrenched in the Linux camp have either avoided asking, or haven’t yet realized was under discussion. But as far back as 2007, academic developers have been building virtual machines — not containers, but Xen-compatible VMs — that replaced separate operating systems with library routines taken from real OSes, but instead compiled into applications.
Grabbing Microservices by the Rump
A 2012 doctoral dissertation by Antti Kantee [PDF], a student of Finland’s Aalto University, chronicles a five-year project to develop a kind of operating system construct called an anykernel. As originally conceived, it was a feasibility study into whether the task-specific drivers relevant to an application being compiled could be merged with the application rather than archived within a monolithic kernel. This way, the kernel could be shrunk down to only the general-purpose routines.
Within the application, the bootstrap that would run the task-specific drivers on behalf of the operating system, was whimsically (or perhaps with other non-technical inspiration) named the rump. It is the evolved form of this very project, still called Rump Kernels, that was at the heart of a demonstration at DockerCon EU in Barcelona last November that may have won over the hearts of Docker Inc. executives. That form, which is only about as old as Docker itself, takes the shape of a redistributable application capable of running itself — one aspect which unikernels and Docker containers certainly have in common.
“What Unikernels allow you to do is basically get rid of the operating system,” said Chief Technology Officer Solomon Hykes in a recent interview with The New Stack, “and instead compile into the application the small bits of operating system that it actually needs.
“In other words, it takes the idea of shrinking the payloads to the absolute extreme,” the CTO continued. VMs are very heavyweight. You have to bundle the entire OS with the machine. Containers shrink the payload, by including only a few OS components. Unikernels shrink that footprint even further.
It isn’t really as if the operating system ceases to exist. More to the point, it ceases to be a separate entity from the compiled application. A unikernel, therefore, is an indivisible unit of computing logic. As a microservice, it carries the promise of unlimited scalability. And as a virtual machine, it is designed to run under the Xen Type 1 (hardware-level) hypervisor.
And as an architecture, there’s a certain elegance in unity. So when, eight years after the “anykernel” was first conceived, the concept first came to light outside of academic circles, competing vendors cited it as the next potential container-killer.
In an August panel on unikernels at ContainerCon NA 2015, Docker contributor Jérôme Petazzoni characterized unikernels as inhabiting a sort of middle ground between containers and VMs, addressing users’ continuing concerns about the uncertain security picture for containers by re-introducing the hypervisor as a kind of safety service.
While acknowledging at the time that unikernels were “interesting,” he did characterize them as addressing a separate set of challenges from containers, effectively distinguishing them to some extent.
A few months later, Apprenda CEO Sinclair Schuller wrote for GigaOm, “Technologies like Docker and CoreOS have done fantastic work to modernize how we consume infrastructure so microservices can become a reality. However, these services will need to change and evolve to survive the rise of unikernels.”
The Would-be Rival Becomes an Ally
Or, they could simply contain those unikernels and put the issue completely to rest.
Last September at The Strange Loop 2015 conference, an infrastructure automation engineer for Boulder, Colorado-based Rally Software named Matt Bajor, entitled “Look, Ma, No OS!” explained in the simplest language yet how a unikernel architecture could eliminate no fewer than five tiers of evolutionary abstraction from the ordinary stack — in such a way that was also perfectly complementary with, not contrary to, the notion of containerization.
After reminding developers of the extensive nature of the modern applications stack, including continuous integration/continuous deployment (CI/CD) platforms, orchestration systems, hypervisors, and configuration management systems, Bajor noted, “These systems work for us, but they are extremely complex. They’re very hard to reason about as a whole, and debugging is difficult. One of the reasons… is it’s almost impossible to replicate the environment in which you’re running, one-to-one.”
Indeed, CI/CD practitioners do make the case that automation of any task, especially a testing procedure, is impossible when the steps needed to replicate that task are so complex and variable that they cannot be reproduced. Containerization has simplified the automation of testing and deployment procedures on the front end of the scale. But after a container enters production, Bajor notes, it becomes such a unique order of beast that reproducing it for purposes of debugging becomes impossible.
CI/CD engineers would argue that it’s much simpler to simply rebuild the container using tighter code, and then redeploy it, rather than try to diagnose it. While that may indeed be an optimum state of affairs, it may also be impractical for containers designed to run legacy software that maintains persistent ties to active databases.
What’s more, Bajor noted, operating systems themselves are conceived, produced, and deployed by a team completely separate from the one building the software. In a typical container-driven environment, it’s the OS that hosts the processes. So when the OS is patched, new possibilities arise for bugs to creep into the process that weren’t present when the code was first passed into production. Once again, CI/CD engineers would argue there are methods for maintaining the proper pairing between deployed code and their respective operating systems, and platforms that enable developers to test and deploy the appropriate patches. Yet even then, these methods are complex — and, Bajor argues, unnecessarily so.
“Patching is not broken down into functionality versus security,” he told the Strange Loop audience. “When we go in and patch a system, we’re going yum update, and applying everything that is there, whether it’s security or functionality. And if it’s not in use, why are we paying the price for it?”
A few months later, the Unikernel Systems team presented its integrated container approach at DockerCon. That team included Amir Chaudhry, who also contributes to MirageOS. Despite its title, MirageOS is not really an operating system, but rather a library of all the things an operating system uses, written in OCaml (Objective Caml) and run on Xen — the modern form of a 20-year-old abstract machine language that’s functional, declarative, and stateless.
In a blog post published soon afterward, Chaudhry made the case that the simplification offered by unikernels and that offered by containers are not only complementary but from one perspective, intended for one another.
“On the one hand, we have the traditional method of placing a full OS stack in a VM with a single application on top. A natural next step is to use containers which run on top of a single OS, giving better resource usage and allowing each application to be more self-contained. When viewed this way, unikernels are just another step on this path and can be thought of as extreme, self-contained applications.”
Integrating the Integrator
The challenge ahead, wrote Chaudhry, is to make unikernels as easy to implement as Docker made containers. Docker CTO Hykes told The New Stack that the goal for his company will be to implement unikernels in such a way as to mask their implementation from developers, effectively rendering the incorporation of unikernels a zero-step process.
“The beauty of it is, if you’re a developer using Docker, basically nothing changes,” said Hykes. Developers would continue to write Docker files (the build manifests) just as they have before. New functionality would be added to Docker, he said, by way of plug-ins. It would be here where extra configuration tooling may be required. “But the user experience of Docker does not change, and the standard interfaces of Docker don’t change. And that’s the beauty of this combination.
“You could say, everything happens at build time,” he continued. “Every time you build your application, you’re building a one-time, unique operating system that does only one thing: run your application. From that point of view, there is very little that we use from the traditional Linux deployment tools. Every application is now completely unique from the bottom up.”
During his September presentation, Matt Bajor listed no fewer than eight strong, healthy, open source unikernel projects going on at that time, throwing in an internal Microsoft project called Drawbridge for good measure. By this measure, Docker’s acquisition move Thursday does not take unikernels off the table as a potential competitor to Open Container Initiative (OCI) containers in some form, the same way an Oracle acquisition into its ongoing platform removes potential future competitors to Oracle.
Nevertheless, the move does prove that any portrayal of unikernels as fundamentally opposed to containers is purely artificial. Docker has not set a timeline for any commercially viable appearance of unikernels in its developer tools, and may yet need time simply to digest the concept. A contained unikernel would have the benefit of integrating well with the existing toolset already under way for Docker containers.
But a self-contained unikernel was already designed for an existing ecosystem: Xen. If Docker makes unikernels popular, it could actually clear the way for someone else to barge in and make unikernels into a market.
More importantly, though, the mere validation of the concept by one of the most innovative vendors in data center technologies today brings the entire issue of operating systems’ continued ubiquity front and center. The first generation of virtual machines were desktop and server operating systems made to “believe” they were running on generic hardware; the current generation of containers are shrink-wrapped applications hosted by an operating system. We’ve grown accustomed to simply reproducing the OS whenever we need its functionality. Now, as we face a future full of microservices, that old custom may not scale as well as we thought it would.
Unikernels are essentially another symbol of the ever evolving story about software development and the continued need to make isolation better and security just something that gets baked into app development and management. Technologists have achieved a degree of compatibility, and the next effort is to get better performance. By bringing unikernels to Docker, it could allow for greater familiarity with the technology. That’s especially true considering the developer tool ecosystem for Docker and container technologies.
Apprenda and Docker are sponsors of The New Stack.
Cover image: “In the Corn Field” by Lars Plougmann, licensed via Creative Commons 2.0.
