Mesosphere’s New SDK: Build Apps on a Laptop, Scale to the Entire Data Center
The new software development kit (SDK) for Mesosphere’s data center operating system (DCOS) gives developers a kind of proving ground for building applications on a laptop that can be scaled up to a whole data center without changing the code. It’s testament to this new idea about the data center developer, someone who can create distributed applications with a command line interface and basic user interface. With the SDK, the developer may use various programming languages, such as Java, Python or Go, to leverage what is essentially a giant computer — pooled resources that abstract the complexities of data center management for the purposes of building applications.
This is a significant development for how we think about the data center and its role for developers. It’s a development that transcends the discussion about containers and puts it in a context about application development and management at scale.
“We want to make it extremely easy to write for this new form factor,” said Florian Leibert, Mesosphere’s CEO, in an interview with The New Stack about the news announced this week.
“You don’t even notice that you’re writing a distributed application. It appears to you as though you’re writing an application just against resources. The boundaries between the machines become really blurry.”
Leibert characterizes Mesosphere’s new SDK as the successor, of sorts, to the Mesos API. It will expose primitives through the API that can enable the automation of container-oriented task scheduling. Though Mesosphere mentions Docker specifically, executives are careful to point out that container scheduling extends beyond just Docker, to cover all kinds of isolation and partitioning since the science began, including straight LXC.
“We’ve always used a form of containerization within Mesos, even before Docker was around,” said Leibert. “For us, containerization is very, very pluggable. … We give you ultimate flexibility in what containerization technology you want to use.
“You can use the appcontainer spec, you can use Docker, or you can use native cgroups. And it’s conceivable that, if in a year or two, another container technology comes out, well, it’ll be really easy to swap it in. We’re really agnostic to that.”
A New Spark Every Month
For new developers to get a handle on how distributed computing works, the SDK will create a completely localized DCOS environment directly within developers’ laptops, converting them into miniature data center proving grounds. Following the SDK’s “Getting Started” track, we’re told, will lead developers on a path where they may eventually submit their first DCOS apps for inspection, and eventually receive Mesosphere certification from the company.
The SDK will bundle a number of developer tools, some from third parties, that facilitate the development of distributed applications. One such tool, Leibert mentioned, is a naming service that enables components of networked applications to more easily discover one another.
The CEO pointed to the success of the Apache Spark cluster computing system, specifically at how quickly it managed to render the MapReduce component of Hadoop effectively obsolete in just a few months’ time. He attributed Spark’s perfect fit with Mesos as the key reason why, and said he believes the Mesosphere SDK will enable developers to produce distributed systems on the scale of Spark, with only months or even weeks of effort.
“Not just Spark, but message queues like Kafka on top of Mesos now, are built against this SDK,” said Leibert, “and they run extremely well, extremely robust. It’s much easier to actually think about writing a distributed system in this fashion, because you don’t have to think about individual machines. Thinking about individual machines and wiring them together is the wrong mental model to write a distributed application.”
Hyperscale-in-a-Bottle
It sounds like an impossibly tall order: Step 1, you learn how to write “Hello, world!” Step 2, you build an analytics-driven container that can be scaled a thousand-fold across a massive data center. The difference, with containers and Mesos, is that the logistics of the distribution never have to be programmed. You build the container, and as long as you use the Mesosphere API, the logistics can be determined later through the orchestrator.
“If you’re building a new, complex analytics system for a vertical like financial services, you’ve got to have a lot of domain expertise, you might have to have some Ph.D. statistics, said Mesosphere Senior Vice President Matt Trifiro. “That’s a fairly complicated system. But if you’re building a new PaaS layer inside your own company (by the way, there are 12 or more PaaS that are written to our stack, half of them built by companies like Apple for their own internal use), that’s a fairly straightforward PaaS to do.
“So I think you’ve got a whole class of applications,” Trifiro continued, “from highly complex and specialized to the relatively straightforward, but solving for a particular niche. … This is probably the largest greenfield opportunity for a developer, from a business perspective, because you have a major transformation in businesses needing to deliver these types of services. Ten years ago, only Google and Twitter had to be Google and Twitter — five years ago, even. Now, everybody has to have a piece of that. Anybody who sells a ‘thing’ connected to the Internet, needs an Internet of Things backend. Everybody who has cash registers needs real-time data analytics. This is the problem that every customer company has to solve, not just the Silicon Valley folks.”
Developers interested in early access to the SDK may now sign up for the Mesosphere DCOS Developer Program.
Docker is a sponsor of The New Stack.
Feature image: “245-Neon-Spark-Blue” by RAVEfinity is licensed under CC BY-SA 2.0.
