Docker Steps into Large-Scale Container Orchestration with Conductant Purchase

Containers are revolutionizing software development, with Docker blazing a trail for not only working with containers at scale, but how applications are shipped, built, and tested. With the rise of container-based solutions, Docker developed cluster-management software Docker Swarm to those working with containers in the enterprise.

On Thursday, Docker took the next step in addressing container orchestration at scale, announcing that it has acquired Conductant, a company that was in the process of building out the Apache Aurora container orchestration software. Aurora is based, at least in its general architecture, on the Borg software, built by Google to manage its own container operations.

To understand the significance of this purchase, it’s worth going into the history behind Conductant, which, until Thursday was still operating largely in stealth mode.

Conductant co-founder Bill Farner worked on Borg at Google, then left to join Twitter in 2009, when microservices were just beginning to come into their own. Managing microservices at scale was a challenge for Twitter, one which made Farner reflect on his time spent working with Google Borg.

It Began With the Borg

What is Google Borg? Simply put, Borg is software designed to manage clusters at scale. It is, in a sense, the father of what is now known as Project Aurora. Built to maximize high availability for those running thousands of machines, Google Borg provided developers process isolation, minimal fault recovery times, and scheduling.

Google Borg is where Farner first saw orchestration platforms centered on high availability clusters at work, taking the idea to Twitter after parting ways with Google. Farner set out to build something similar for Twitter over the next six years, teaming up with Apache Mesos along the way.

Apache Mesos Internal Infrastructure

The engineering team at Twitter quickly realized the benefits to using Mesos as a part of their new cluster management platform. On-boarding Mesos helped better streamline tasks such as orchestration, jobs, deployments, and, of course, scheduling. By focusing on operations level development, the team was then able to use Mesos to its full potential as the glue of their new platform.

The result was the aforementioned Project Aurora, a full-fledged cluster management system for those running services at scale. Shared as open source software in 2013, Apache Aurora boasts a thriving development community helping to improve and build on the platform.

Scalability was a key focus for the development team at Aurora though Farner noted in a post on the Docker Blog that defining the difference between technical and organizational scale is key. Aurora now powers hundreds of thousands of clusters at Twitter, with Farner taking the project to the next level.

Task Life Cycle Apache Aurora

In 2015, Farner co-founded a company to commercialize Aurora, called Conductant, with David Chung and John Sirois. Chung’s prior experience at Google and Zynga, coupled with Sirois’s own time at Google, led the trio to formulate a goal of streamlining the process for shipping code quickly without requiring a bloated team of managers as a project scaled.

When deploying microservices at scale, flexibility, speed, and ease of deployment are crucial. These factors were some of the building blocks which led the team at Conductant to re-imagine an orchestration platform on top of Docker — One which could bring together technology from not only Google Borg, but Aurora and Mesos to help developers manage their microservice fleet running on top Docker.

Dreaming Big with Docker

Since Thursday’s announcement, Docker is wasting no time in coming up with ideas for how to integrate Aurora into Docker—Particularly Docker Swarm. With Aurora and Docker Swarm as part of one’s infrastructure, enterprises could manage microservices with the power of both platforms. As such, this will provide stability, ease of deployment, and much more to Docker users.

Sample Integration of Apache Aurora with Docker Swarm, via Docker

Farner, Chung, and Sirois continue to work on operations-focused development with the recently-announced Docker Cloud platform. For those interested in learning more about deploying containers on a larger scale, Docker is hosting Swarm Week starting March 7th, 2016, A week-long event which will help developers and engineers learn more about using Docker Swarm as part of their infrastructure.

As more companies continue to develop applications at a breakneck pace, the need for high-availability orchestration solutions across one’s container infrastructure is clear. We’d love to know what solution you’ve found to work the best for you. Take our Container Orchestration survey and let your voice be heard.

Docker is a sponsor of The New Stack.

Feature image via Pixabay, embedded images via Docker, Apache Aurora, and Apache Mesos.