Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
CI/CD / Containers / DevOps / Kubernetes Aims to Soothe the Savage IT Shop with a DevOps Platform

Jun 17th, 2016 9:14am by
Featued image for: Aims to Soothe the Savage IT Shop with a DevOps Platform aims to sooth the raging complexity that seems to be inherent in maintaining modern applications, through what it calls a “DevOps Automation platform.”

Not only taking on the challenges of developing, provisioning and deploying applications, automation also has to address all the software components that have to be managed independently, as well as the various parts managed by various people and address the underlying storage and network as well, the company believes.

“While the whole Docker movement has been very good for developers, it’s the operational tooling around that that’s still evolving. That’s where the conversation is shifting from Docker to things like Kubernetes and all the tooling around, ‘How do we actually put [containers] in production and manage them?’” said Jasnoor Gill, Calm’s product director.

“That’s been an extremely good conversation starter for us.”’s Jasnoor Gill, on the trend of containerization.

The whole Docker movement has been a boon for companies like Calm because it shifted the conversation beyond just code to runtime as well, according to Gill.

He says the company isn’t competing with continuous integration/continuous delivery vendors. Yes, it becomes part of the pipeline, yet “The biggest problem a lot of the CI/CD tools and DevOps tools have is that it only concerns itself with orchestration,” Gill said.

“While that works for architects, developers, and DevOps, it doesn’t always work for IT, and it definitely doesn’t work for the CFO. There’s no way to apply your enterprise policies. If I’m writing a script, I might add some code that would fire off an email at a certain point in my script to get an approval, but what you’ve done is mixed your application policies with your business policies. But you might have a different set of policies for Dev, for staging and for prod. That means you have to maintain three different types of scripts for each type of stack that you have,” he said.

Calm offers a way to create blueprints to declare application topology.

And current CI/CD solutions lack “last-mile” automation in the application lifecycle, he said.

“We see the CI/CD and the pure environment as one layer, but you also need a complete layer for cost control, health checks, audits. No [CI/CD] platform gives you all of that in a single solution.”

Calm is an automation platform for hybrid cloud infrastructure and microservices-based application architectures. The idea is to calm IT management by addressing challenges created by the emerging IT landscape at the same time companies are dealing with their legacy infrastructure.

What’s in the Box?

Calm offers:

  • Model-based automation in which flexible service definitions are decoupled from the specific environment, as are IT policies.
  • Full-stack, workflow-driven orchestration in which tasks are coordinated centrally across diverse infrastructure, multiple containers, and various supporting resources.
  • Automated lifecycle management.
  • Rolling and canary updates without the need for code or scripts.
  • Auto-scaling based on event policy.
  • Monitoring for failures and self-healing not only the node but including upstream and downstream dependencies.
  • Allocating resources and enforcing team budgets.
  • Automated access and permissions policies.
  • High-resolution auditing and tracking.

“What we realized is that it was not very easy to automate, set up and manage cloud resources,” Gill said. “If I want to set up a 50-node cluster for performance testing, it would take a couple of hours just to set up that cluster, even if they had scripts or Chef automation to launch them on AWS, hook them all together.” offers the ability to orchestrate multi-tier systems offers the ability to orchestrate
multi-tier systems

At the same time, many organizations have machines running that no one seems to know what they do, but they’re afraid to shut them down for fear something else will break.

“But with public cloud, every time you have resources running that you’re not using, you’re costing someone a big bunch of money. So that’s where we started from,” he said.

“With Calm, we let you automate all of this. We let you deploy code. We let you use your existing Chef, Puppet, Salt recipes. We provide an interface to do that,” Gill said. “From an IT perspective, we take care of all the objects, the compliance, the cost controls on top of your hybrid cloud. You want to give your developers access to all these resources, but you still want to enforce certain IT policies, cloud control policies in terms of cost.”

Calm supports bare metal, VMs and Docker orchestration. It offers public cloud support for AWS and Azure and is looking at SoftLayer, Digital Ocean and Google Compute to support next. It also supports the private clouds of VMware, OpenStack, Xen, KVM as well as the Nutanix Acropolis Hypervisor. can be used to automate playbooks. can be used to automate playbooks.

On the development side, Calm supports Chef, Puppet Saltstack, Jenkins and other tools through APIs.

“What that means is that if you have any automation with any of these existing tools, we can just click onto that automation. You don’t have to rebuild your tooling,” Gill said.

“We apply IT policies on top of it, so we take care of cost control, we take care of expiring workloads that no one needs anymore if someone has to sign off on something. This is stuff that’s been done before, but in the past, it’s been done outside the tooling. You have to send an email or call someone [to get approval]. Combining all those scripts into a single system has been the challenge, and that is what we’ve tried to solve.”

Calming effects

Calm is a spinoff of the India-based automation company Idea Device. The company is about 2 ½ years old; it released its product about a year ago. Its release of version 1.7.1 in April included Microsoft Azure integration and the support for LDAP to manage users and groups.

With two employees in San Francisco, it’s just now expanding in the United States.

Its customers include the National Stock Exchange of India, online marketplace Snapdeal, fashion platform Voonik, mobile payment app Chillr, data-protection vendor Druva and the Groupon for India Nearbuy.

Milind Borate, chief technology officer of Druva, said that thanks to its KVM environment, Calm has helped reduce time to deployment from a day to 15 minutes.

Docker is a sponsor of The New Stack.

Feature Image: “Serenity” by Brandon Atkinson, licensed under CC BY-SA 2.0.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack, Docker, Calm.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.