Docker on Diego, Cloud Foundry’s New Elastic Runtime
Cloud Foundry has a new elastic runtime, written in Go and now with support for Docker as part of Diego, a new orchestration manager that distributes tasks and application processes.
Diego serves as an app execution, container-based pluggable scheduler and health check manager, said Andrew Shafer, senior director of technology at Pivotal. IBM, SAP and Cloud Credo are also helping develop the open source project. The project is open to anyone. See more on Github.
How Diego fits with Docker demonstrates an effort to make Cloud Foundry more accessible and streamlined. Diego also helps bring a cleaner container abstraction with Warden, the container technology developed originally for Cloud Foundry.
The demo covers:
- Deploying Docker
- Scaling from 1 to 300 instances in less than a minute
- Balanced placement with distributed auction
- Log aggregation
- Health checks
- Zero bad requests when 20% of the infrastructure disappears
- Respawning the missing instances
“You can drop a Docker image into Diego and it will mount all the layers using a Docker library,” said Phil Whelan of Active State.”It will run in a Warden container. Diego does not run Docker containers. It has an interface to run compatible images in Warden containers. For the end user, there is very little difference. You can push a Docker image into Warden. Diego is designed to run different backends.”
Easier Management
Diego has allowed the Cloud Foundry team to simplify the platform as a service (PaaS). The droplet execution manager (DEA), health manager and some of the cloud controller are now taken care of by Diego, Whelan said. Cloud Controller is not part of Diego and remains in Cloud Foundry. For the developer, Cloud Foundry will run the application processes and distribute them to Diego, which then manages the tasks and longer processes across its cell network. It makes Diego more closely resemble its own microservices environment, granted with still much to develop. Developers who don’t want or need all the features of Cloud Foundry could still use Diego as a core component to build a PaaS for their needs, Shafer said.
“The cloud controller was in charge of everything before and had to manage the placement metadata for the whole deployment,” Shafer said in an online chat.
The ability to create a PaaS out of Diego follows a movement that is fast developing in the Docker ecosystem. For example, New Relic had never before used a PaaS. But with Docker, they built their own homegrown service, which they use for deployments. Granted, building a PaaS with Docker does require a supporting ecosystem which would include the need for robust orchestration.
Cloud Foundry’s integration with Docker shows how fast the new container technology has found acceptance, Whelan said.
Docker is important for the future of PaaS. Every new PaaS has a strong connection to Docker. To ignore Docker containers ina PaaS would be suicidal.
So what will become of Warden, the Cloud Foundry Docker technology? Very simply, it will accept Docker images to run in Warden itself.
Distributed architectures are by no means easy systems to build. Orchestrating all the disconnected components to make a coherent service is complex, made even more so with the assumption that failure will happen. Servers may go down or the network slows. Historically, companies would architect the network in advance. But more modern practices use collaborative approach with short cycles and feedback loops instead of up front planning that is often “divorced” from reality, said Onsi Fakhouri, Diego’s product manager in a post by Whelan last week on the Active State site.
The first few weeks of Diego’s development were spent understanding the scope of the domain and the problem at hand. They drew up a simple vision for the broad outline of Diego’s architecture, Fakhouri wrote. The team also built a comprehensive test suite to validate that its components work together. The orchestration happens with a centralized script that defines the coordination.
It is the orchestration that is most relevant in terms of what Diego offers. The scheduler algorithm orchestrates where containers go and hence where application instance processes are run, Fakhouri said.
The orchestration problem that Diego resolves makes it comparable to platforms such as Google Kubernetes and Mesos. Kubernetes and Diego both run on etcd, the configuration technology that is also what CoreOS uses to manage its OS for server deployments:
Mesos is simply a communication infrastructure and protocol for enabling scheduling across distributed resources. Diego does that (admittedly with more specificity) and a lot more: containerization, log aggregation, routing, health-management, etc. Frameworks built on top of Mesos such as Marathon are more directly comparable to Diego.
Mesos now treats Docker as a first-class citizen.
As for Kubernetes, it solves the problem of generically coordinating containers. In comparison, Diego specifically supports app deployment, but is heading in a similar direction “without sacrificing its ability and mandate to cater to CF,” Fakhouri said in the interview.
But why the name Diego?
“Cloud Foundry version 1 and version 2 containers run in a service originally called DEA,” Shafer said. “So they were rewriting the DEAs in Go.”
Get it?
DEA-Go became Diego.
Ahh, those clever geeks.
