A Docker Fork: Talk of a Split Is Now on the Table
Discussions about a split from Docker are now underway among several Docker ecosystem vendors and end users. Expressing frustration of Docker’s management of Docker Engine, the technologists with the companies are exploring ways to address various issues around supporting enterprise Docker deployments.
Several options are under consideration, including the possibility of forking the open source Docker engine altogether. According to a number of sources close to recent discussions, representatives are involved from companies such Red Hat, Google, CoreOS, Huawei and two large end-user customers.
Spokespeople from Red Hat and CoreOS denied any knowledge of Docker-related talks. Docker declined to publicly offer a comment. Google and Huawei have not replied to requests for a comment.
It’s a seminal moment for the container ecosystem. None of the technologists with these companies want to fork but they have also voiced concern over the Docker codebase’s lack of stability, which can be problematic when building out additional services and supporting customers relying on Docker in production.
Then again, many of the companies participating in the talks are also using containers for building out large-scale distributed systems, and could see Docker’s recently-launched built-in orchestration capabilities as a threat to their own orchestration products, nearly all of them based on Google’s Kubernetes.
“What’s happening right now, if we are not careful, will fragment the container ecosystem, and it make it impossible for single containers to target multiple runtimes,” one observer on Hacker News pointed out.
— The New Stack (@thenewstack) August 25, 2016
An Enterprise Issue?
A worry we’ve heard repeated from the Docker ecosystem is how Docker’s aggressive release schedule puts third-party system providers at odds with their own customer base.
“Docker consistently breaks backend compatibility,” said Rob Hirschfeld — CEO of RackN, Kubernetes community leader (and occasional TNS contributor) — who was not involved in the talks. RackN’s application lifecycle management platform both uses and provisions Docker components, so changes to the backend have business impacts for supporting customers.
Docker uses its Docker Engine as a product, not as a component that the community uses to build out its own services, Hirschfeld charged. This product-based approach means that the operator is forced to fix backend compatibility that breaks when Docker introduces new innovations or merges components like Swarm into the Docker Engine.
“While we use those features, the changes lead to problems with stability, versioning, migrations and backend compatibility,” Hirschfeld said.
Until last week, the conversations about dissatisfaction with Docker have largely been private. But now the talk is moving into public forums as exemplified by these tweets from Google’s point person for evangelizing Kubernetes, Craig McLuckie:
.@docker must be allowed to innovate and profit (they have legitimately done amazing work!), but we need boring core infrastructure today
— Craig McLuckie (@cmcluck) August 26, 2016
It really is time for a container runtime and format standard to emerge beyond the (current) scope of OCI.
— Craig McLuckie (@cmcluck) August 26, 2016
The long-simmering frustration has culminated with Docker, Inc’s recent inclusion of Docker Swarm in Docker 1.12. With Swarm, Docker Engine allows users to manage complex containerized applications without additional software, using the same command line structure and syntax that developers are familiar with using the Docker containers. The Docker orchestration capabilities are opt-in; they must be activated by the user. Though not opting in may lead to backward compatibility issues down the road.
Previously, these orchestration capabilities, as they are known, have only been offered through third-party tools such as Kubernetes, the orchestration platform open sourced by Google.
Although Google does not directly sell enterprises container orchestration software, its open source orchestrator does provide an easy on-ramp to Google Cloud Platform services for containers. Other companies, such as CoreOS and Apprenda, do offer commercially supported versions of Kubernetes. Huawei is also launching a Kubernetes-based container engine.
So, not surprisingly, all of these parties are eager to move the conversation from containers to container orchestration instead.
Bob Wise, the Samsung SDS cloud engineer who leads that company’s Kubernetes involvement, has also called for Docker to slow its roll on container innovation. Samsung Electronics is in the process of acquiring Joyent, which offers cloud-based container support services.
“If your team is working deeply in Kubernetes, Mesos, or Cloud Foundry, you need a stable, simple, boring container implementation with minimal essential characteristics, and community agreement around image creation, naming, and publishing,” Wise wrote in a blog post last week. “You need to use the same simple, boring container implementation that everybody else is using. As a community, we need to slow the pace of change of the fundamental building blocks. Stability will allow the systems built on top to thrive.”
— Joe Beda (@jbeda) August 26, 2016
Likewise, Apcera technical product manager Phillip Tribble expressed in a personal blog post — in the most diplomatically way possible — for Docker not to trumpet newly launched features as finished enterprise-ready products. “It is not wise to explode the Internet and conventions with marketing material about exciting new features that do not work as presented,” Tribble wrote.
Apcera’s business model is based in part on its support of Docker containers.
Tribble’s post inspired others to express their own experiences about Docker on Hacker News, including a few bemoaning bugs around new releases. One reader talked about spinning up 70,000 to 90,000 containers a day, and having trouble with about 9 percent of them due to “diverse Docker bugs.” That number had been lowered to 4 percent after the latest update.
But others are praising the software’s stability. “We’ve been using Docker in production for about 3 years now, and have not had any big issues,” another reader commented. “It’s a neat wrapper over some cool Linux kernel features.”
In fact, Docker does have a large following of supporters who are of the opinion that Docker’s software is stable. Docker engineers are also quick to boast of the company’s solid foundation.
— Arnaud Porterie (@arnaudporterie) August 18, 2016
In any case, Docker seems to have no intentions of stifling its own innovation for the sake of its potential competitors, as shown through a recent Twitter dispute between Google Kubernetes evangelist Kelsey Hightower and Docker Chief Technology Officer Solomon Hykes over the role that the Open Container Initiative’s (OCI) image and runtime standards should play in the standardization of the Docker-based container stack.
OCI was formed in order to provide exactly what folks like Wise are calling for: boring standardized container tech.
Yet Hykes cast doubt on the extent of OCI’s usefulness for third-party providers. Hykes noted that those providers who claim to offer full support of Docker containers without the Docker Daemon (presumably through runC runtime engine) are actually only offering “pseudo-support” of about 90 percent of Docker’s features.
“All I know is Docker moves fast, and as a result products claiming ‘Docker Support’ are lying,” Hykes wrote. In another Tweet, Hykes went as far as to dismiss the OCI image format as a “fake standard.”
— Kelsey Hightower (@kelseyhightower) July 29, 2016
The problem is further exacerbated by Docker, Inc.’s hold on the Docker trademark. The trademark means that companies may face liability if they use patches, code or packages not approved by the Docker community. In that case, a company may be unable to support its own customers due to patches not getting approval from Docker or upstreamed into the project.
The result: third-party vendors are forced to live by any decision Docker decides to make about the open source project and wait for Docker’s patches whenever something breaks. The best outcome would be some agreement with Docker to guarantee stable releases.
One's feature is another's bloatware > container run time fork (of Docker!) brewing as ecosystem wants stable & small building block.
— Rob Hirschfeld (@zehicle) August 27, 2016
Also at issue is which features should be implemented by Docker and which should be handled either by the underlying OS or another component in the stack.
Another company relying on Docker, but also competing against Docker, is Red Hat. In a number of conferences this year, including LinuxCon North America 2016, Red Hat engineer Dan Walsh describes Red Hat as being stuck between its customers’ increasing use of the systemd to initialize Linux systems, and Docker’s seeming reluctance to work with systemd, instead relying on its own Docker Daemon to provide features around initialization, service activation, security and logging for containers.
“For the last three years, we have been trying to get good integration between the systemd and Docker, and what I have found is that two leaders of the two efforts are very strongly opinionated,” Walsh said, referring to Hykes and systemd creator — and current Red Hat employee — Lennart Poettering.
“So, when I bring up the machine, who brings up system services, systemd or Docker?” Walsh asked. A badly implemented system can have systemd and Docker Daemon fighting one another.
For its customers, Red Hat maintains its own forked version of Docker, which include patches that Docker may one day incorporate, or may never choose to include.
Options on the Table
Time will tell if these informal talks lead to a more coordinated effort. There are options that may suit all parties. For example, there may be a stable version of the Docker image format and runtime that is managed through the OCI. This would allow ecosystem partners to have stable components and allow Docker to continue with its own Docker Engine. In this scenario, the OCI runtime would be a fork of Docker.
But if the ecosystem companies want a complete break from Docker then that would mean a complete fork with a different name and need for a new group to manage it all. CoreOS, for instance, offers an alternative “daemonless” container technology, called rkt, though it has not seen significant market traction as of yet, at least not to the level of Docker’s success.
This is new ground, signaling how fast the market is maturing and the stakes at play. Exploring the container ecosystem from a political and economic perspective is important in helping better understand the technical direction of the community. That’s certainly true today in face of the deep divides between Docker and the container ecosystem and the active discussions taking place about forking Docker or, if nothing else, creating a stable alternative that would allow for a standard core that could be built upon and assure stability for end users.