What news from AWS re:Invent last week will have the most impact on you?
Amazon Q, an AI chatbot for explaining how AWS works.
Super-fast S3 Express storage.
New Graviton 4 processor instances.
Emily Freeman leaving AWS.
I don't use AWS, so none of this will affect me.

Deis Aims to Extend Kubernetes into a Platform

Nov 18th, 2015 1:32am by
Featued image for: Deis Aims to Extend Kubernetes into a Platform

In just a few short months, Google’s deft move to build an open consortium around its Kubernetes orchestrator has shifted the platform focus away from containers, and onto container orchestrators.

Perhaps the biggest indicator of that shift came last week at KubeCon in San Francisco, where Deis — now the brightly polished new division of Engine Yard — unveiled a package manager for workloads called Helm.

Helm will take a wholly new approach to deploying workloads in a PaaS-like manner, with Kubernetes — not Docker — as the objects of these deployments, according to principal architects Jason Hansen and Matt Butcher. The Deis team saw that no two data centers or cloud environments work alike, so no technique for deploying one type or even one identity of workload will translate perfectly between platforms, like a standardized template.

So Helm will literally take a crowdsourced approach to the problem, enabling users to experiment with deployment patterns for particular workloads, while filtering through to the common patterns between successful deployments. Those patterns will effectively create up-to-the-minute best practices, that Helm users can utilize to their advantage over time.

The Community Contract

“The idea is, people can collectively define what the best practices are for deploying apps into Kubernetes,” explained Platform Architect Butcher, “by building these Kubernetes manifest files — which are the contracts. And at the same time, while they’re defining these, they’re building things that are useful — an NGINX replication controller definition for anybody who wants to use that, or a Postgres one for anybody who wants to run the database.

“And it becomes sort of an experimental ground for the microservice idea,” Butcher continued. “If we do it right, then we’re building contracts that allow these things to be assembled in almost arbitrary, surprising ways.”

Kubernetes first and most general task, he stated, is compelling devs and DevOps to figure out how to get stuff to run in Kubernetes. The Deis team knows that better than anyone: They’re actively reworking Deis version 2 into a system that runs atop the Kubernetes platform.

So they first approached this problem of Deis “getting stuff running” on Kubernetes in the context of getting Deis itself running there. It was through this process, Butcher said, that his team realized that developers and DevOps professionals were missing a kind of bootstrap process to get the first applications — the ones most everyone needs — deployed to Kubernetes.

“Number two, it would be nice if we had teams who were working together to have a tool that helps them collaborate on developing this stuff for Kubernetes,” he added. “The deeper we got into this, the more we started talking as if we were talking about package management — like maybe Homebrew for OS X, or apt-get or yum for Linux. And we realized that essentially, that was our vision for how we thought we could help individual developers learning, and those teams, make it over some of the common hurdles that they were experiencing with Kubernetes.”

Jason Hansen and Matt Butcher, DEIS: At Kubecon

Listen to the TNS interview with Jason Hansen and Matt Butcher.

The Semi-definitive Chart

In a Deis blog post last week, Butcher demonstrated the basic principles of running and utilizing Helm. After being run for the first time through the Linux command line, Helm updates and configures itself for the host it’s running in, and then creates basic configuration files based on that host. Configuration files for applications packages are called charts, to continue the seafaring metaphor — although they’re not actually visual in nature, but rather YAML files.

Charts are not default manifests for applications, but rather the embryos for what will eventually develop into their configuration files. While charts are stored in a central repository on GitHub, each one is designed to be brought into a workspace, where it is adjusted for the Kubernetes platform to which its application will be deployed.

“Think of Helm as a complementary Kubernetes tool that paves a predictable, collaborative, version-maintained path for your DevOps team,” wrote Butcher. “The Helm Workspace is your playground. Not only can you modify existing charts, but you can build your own there. You can add version control and track your workspace as its own repository. And we even give you the tools for creating and submitting charts!”

Hansen described a Helm chart as containing metadata about an application, plus descriptive parameters about its specific version, and potentially multiple manifests — for instance, a pod manifest, a replication controller, or a service definition — and the known resource locations for constituent files. Arbitrary labels may be defined for components contained in a chart.

“Application developers have a hard enough life as it is,” said Butcher. “And then there’s the quintessential problem, that I call, ‘throwing it over the wall,’ where you have the DevOps people who are responsible for running all of this stuff in production, and you have the developers who are responsible for building it, and there’s always this handoff process that, all too often, becomes throwing something over the wall.”

When developers build containers, Butcher said, there’s a certain assurance level — provided by the abstraction layer — that those containers will run much the same way in the production phase as in the development phase. This already alleviates much of the headaches among DevOps, who understand the basic instrumentation of containers, but end up doing less of that. Containerization already provided this assurance level, but Helm could help to formalize this relationship further, by presenting it from team to team as a kind of contract — one that isn’t “thrown over the wall,” but that instead blows right through it.

Hansen said that his team’s design goal for Deis version 2 is “to expose as much Kubernetes as we can without wrapping it.” By that, he means providing an easier means for devs and DevOps teams to access the orchestrator, without shrouding it in another process, or embedding it in yet another platform.

Pattern Matches

“Helm is sort of an active experiment,” Butcher said. “If the community together comes to bear on this, and we define a good pattern for running MySQL and a good pattern for running WordPress and HAProxy, all inside of Kubernetes, then if we do it right, and if this whole microservices architecture is all that we are hoping it’s going to be, then people should start pulling these components out of Helm, or using Helm to install them into Kubernetes, and build up some fairly good applications with much less work than they would otherwise have to do, if they were assembling each of these things by hand.”

“Beyond that, it’s really a question to the community, really: What contracts do we want to establish as a community of Kubernetes operators?” said Hansen. “How do we label stuff? We’ve got a couple of conjectures, one being that the responsibility for a service definition, in the context of Kubernetes, is on the side of the consumer rather than the producer. That’s one question that makes application configuration much easier for us, but is also an idea that we want to see validated by the community.”

It’s Hansen’s way of admitting not only that the question of establishing labeling and tagging specifications has not yet been determined, but that it may not be Deis’ or Engine Yard’s role to determine them. Hansen and Butcher genuinely believe that, as long as Helm provides developers and DevOps professionals with an adequate mechanism for customizing their own workload handling procedures, they’ll come up with a more-than-adequate solution to the issue for themselves.

By handling the matter this way, Butcher suggested that Helm could bring devs and DevOps closer together — perhaps even grafting or merging the two teams together. “One of the things that I think we’re passionate about, that I think a lot of people in Kubernetes are passionate about,” he said, “is that we’ll figure out some sort of community, de facto method of using some of these labels, so we can talk about them in a more meaningful sense than if they were just completely arbitrary tags.

“If Helm is going to be successful,” he added, “part of it will be contingent upon people starting to have that particular discussion.”

The New Stack Editor-in-Chief Alex Williams contributed to this story.

Docker is a sponsor of The New Stack.

Feature Image: “Dutch Ships in High Seas off the Dutch Coast” by Ludolf Backhuysen (1630 – 1708), courtesy The Walters Art Museum, used under Creative Commons license.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Coast, The New Stack, Docker.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.