Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.

Mesosphere Now Supported on DigitalOcean For Scaling and Managing Apps Without DevOps Hell

Oct 28th, 2014 8:17am by
Featued image for: Mesosphere Now Supported on DigitalOcean For Scaling and Managing Apps Without DevOps Hell

DevOps is a pain in the ass. It will disrupt your organization and force you to deal with the hell that it is this:

To alleviate the pain that comes with scaling, DigitalOcean today announced support for Mesosphere as a way to abstract the increasing complexity that comes with Internet-scale apps, said DigitalOcean Co-Founder Mitch Wainer. Mesosphere is the orchestration service with $10 million in investment from Andreessen Horowitz that is based on Apache Mesos, which Twitter originally developed to automate its infrastructure. Today Apache Mesos is available to the general public. The Mesos software is also used by eBay, Airbnb, Netflix and HubSpot.

Mesosphere is now available to all DigitalOcean users. DigitalOcean is offering a developer cluster for those who want to first give it a try. Its standard offering is the high available cluster.


It is then a matter of entering your public SSH key and client ID before launching a cluster.

As Nancy Gohring wrote on the New Stack, Mesos manages clusters of machines, automatically scaling apps as needed. It requires a small bit of software on each machine – the software uses zero processing power and “negligible” memory, which coordinates with a master scheduler. The software on each machine reports information about the capacity of the virtual machine or bare metal server to the scheduler, which allocates jobs to available machines. It can automatically scale a variety of jobs including Hadoop databases, nodes running Ruby on Rails, and Cassandra.

Here’s DigitalOcean’s view of the Mesos architecture:


Mesosphere is betting on DigitalOcean as a provider that appeals to the growing developer-oriented business who would use Mesosphere to deploy highly scalable services that would not need to be refactored at some later point, said Florian Leibert, co-founder and CEO of Mesosphere.

Mesosphere and Mesosphere’s Marathon, both of which are installed by default, provide an abstraction that lets developers deploy and scale any Linux workload in a PaaS-like environment without any of the typical PaaS overhead.

It is expected that the first order of business will be to run traditional web-scale workloads, including the typical Ruby, Node, Python, and Java apps, and API’s that usually drive web services and mobile devices, Leibert said. “But we also expect to see a big uptick in these same developers running analytics workloads, such as Spark and Storm, on the same cluster as their web apps.”

In Gohring’s previous story, Jay Lyman, an analyst at 451 Research, said Mesos might be most appealing to businesses that are trying to essentially create an AWS-like environment internally. AWS offers some tools for automatic scaling. But many businesses are still shy about running everything on public cloud infrastructure. At the same time, they don’t want to block their developers from taking advantage of the capabilities available in public clouds like AWS. They’d like to make those capabilities available on their private clouds.

But the new companies of the world are all starting with cloud services and auto scaling is high on the agenda for Mesosphere.

Autoscaling of a cluster is something that a lot of customers have asked for, and the DigitalOcean APIs make this something that can be added pretty easily, Leibert said. They also expect to deliver more fully-packaged stacks that cover major use cases, such as ElasticSearch, Kafka, HDFS, Storm and YARN on the same Mesosphere cluster alongside web- and mobile-oriented workloads.

DigitalOcean is a sponsor of The New Stack.

Feature image via Flickr Creative Commons.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.