Mesosphere’s Data Center Operating System Will Soon Offer a Managed Kubernetes
DC/OS 1.10 was released in Beta form in September, and the pending production-ready general release will improve the overall resiliency, scalability and security of DC/OS, said Tobias Knaup, co-founder and chief technology officer of Mesosphere.
Mesosphere wants the experience of running Kubernetes within DC/OS to be identical with running Kubernetes on a hosted service, such as the Google Kubernetes Engine. “We are really focusing on the operator experience here,” Knaup said. The good news is that users don’t have to be locked into a particular provider to enjoy these benefits.
Kubernetes installs and upgrades will be able to be carried out with a single click. And each Kubernetes deployment will be highly-available by default, with each instance running on three master nodes, along with three etcd servers for service discoverability and three front-end API servers. Should one of the components, or one of the machines go offline, then operations will be automated shifted over to the other, still-working copies.
For security, the Kubernetes deployments use the built-in DC/OS certificate authorities, deploying TLS by default. It has also been integrated into the built-in DC/OS load-balancer, eliminating the chore for those not using a managed K8s cloud service where load-balancing is not supplied as a feature. Multi-tenancy is also supported. You can run multiple Kubernetes clusters together in the same DC/OS deployment, and multiple departments can enjoy their own clusters.
DC/OS will use the stock “upstream” version of Kubernetes, so kubectl and other Kubernetes support tools can be used with the software, Knaup said. The company integrated all the DC/OS services (or applications that configured to run on DC/OS) so they can be easily accessible to Kubernetes. “If you are running Cassandra or Kafka on DNS, we are setting up DNS host names and service discovery so you can very easily connect to those from a pod running on Kubernetes,” Knaup said.
Sign of the Times
In many ways, DC/OS is software for the times. Enterprises are no longer just building applications for internal users. Most all need user-facing applications of some sort. They need to be able to ship apps and updates quickly as well as establish and maintain a set of data services that their apps can share.
DC/OS allows them to run all these applications on a single platform. DC/OS operates as the platform layer, without the developers worrying about the underlying infrastructure. Though unlike most PaaS packages, DC/OS is not particularly opinionated about how you configure your services. Nor is it tied to a particular service provider.
DC/OS is based on Apache Mesos, a platform for running any kind of distributed system. Ben Hindman and the other developers who created Mesos at University of California Berkeley, predicted that, in the future, organizations would be running a lot of distributed systems.
As a result, they created a model called application-aware scheduling, which captures the process needed to boot, or upgrade, or repair, a complex application such as Hadoop. They broke down the problem of “scheduling” an app (i.e. determining what node it should run on) into two parts: One is resource management, which matches pools of machines with applications that are requesting specific resources. The other component is the operational logic, or runbook, for the app itself.
With this approach, the services are so resilient that the admin can play a game of “chaos monkey” and destroy the primary name node, and the system will recover automatically, Knaup asserted.
Currently, DC/OS has over 100 supported data services, which end-users can mix and match to build systems such as the SMACK Stack. A lot of DC/OS users are using the platform for managing data-intensive applications, ingesting data in real-time, Knaup said. Many use Kafka to ingest the data, and Spark to process the data, and stored in a large-scale database such as Cassandra. Many the applications themselves, many written in Akka or Node.js, are managed in containers.
Mesophere plans to launch a certification process for marking those data services that are fully tested and supported.
In addition to Kubernetes support, DC/OS 1.10 brings a lot of other additional features for better managing containers. One is networking. The DC/OS software development kit has been updated to work with the Container Network Interface (CNI), a Cloud Native Computing Foundation-supported interface for Linux-based containers. DC/OS ships already with its overlay network that supports CNI, but this new inclusion of CNI will allow customers to use others, such as Project Calico, and other CNI-supported overlay networks.
DC/OS now also supports the running of Docker images, another feature introduced last year. “Developers can use our tools to create Docker images, store them in a registry, but when they run it, they can download it and substantiate it through the Universal Container Runtime,” Knaup said.
The Health checks that DC/OS executes before and after the upgrades have been improved. Data services can also be provided with non-disruptive upgrades, assuming it is possible to do with the application itself.
A load balancer, HAProxy, has been added to the package, though users can swap in other proxies, such as NGINX. When new containers are rolled out, or the workload scales up, the load balance adjusts automatically. Different load balancing pools can be created for different workloads, such as for production and dev testing.
On the security front, users can now bring their own certificate authorities (CAs) that are needed to manage the authentication process. It supports file-based secrets, needed for running a lot of applications such as Spark and Kubernetes.