Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
Cloud Services / Microservices / Software Development Translates Cloud Costs to Developers

Harness released a product it is calling "Continuous Efficiency," which it says enables developers and DevOps teams to manage the cost of their containerized applications and microservices running in the public cloud.
Jun 16th, 2020 9:53am by
Featued image for: Translates Cloud Costs to Developers, the company that started off with a focus on providing continuous delivery (CD) has set its sights on a new problem: that of wasted cloud spend. The company has released a product it is calling “Continuous Efficiency,” which it says enables developers and DevOps teams to manage the cost of their containerized applications and microservices running in the public cloud, and which Harness Chief Marketing Officer Steve Burton says provides visibility into a realm often obscured.

“Developers are deploying continuously to the cloud, they’re onboarding new microservices, and they have no visibility into the cost of doing that because every 30 days it’s typically a finance team or a cloud center of excellence that is managing that bill and that cost. The feedback loop back to the consumers of the cloud, to us, is broken today,” said Burton in an interview. “Especially around the new technologies like Kubernetes and ECS where it’s containerized, developers are requesting resources and that they get no visibility into the utilization of those resources, as well as the cost.” has evolved alongside the move, for many, from monolith to microservices, offering a faster and easier way to build CD pipelines without spending the time it might take to extend Jenkins scripts to perform the task manually. The company offers a variety of methods, from wizards to YAML to GitOps, to simplify the process, and has since introduced “Continuous Insights,” to provide visibility in the form of metrics, such as mean time to repair (MTTR), lead time, and deployment frequency.

With Continuous Efficiency, the company has shifted visibility to the business’ bottom line, effectively “shifting down” the responsibility of managing cloud spend to those who have the most ability to do so — the developers and DevOps teams. A big part of this shift is providing insight into cost at a level that developers can actually use, and breaking down metrics into pieces that reflect the application architecture, said Burton.

“What typically happens is a CFO or a finance team goes to the engineering lead and says, ‘Hey, why is our cloud bill increasing month over month?’ And when developers look at the bill, it’s in the context of the infrastructure and the services that are being consumed, it’s not in the context of their application or their microservice or their Kubernetes cluster,” explained Burton. “There’s a big disconnect on understanding the cloud bill because these bills are complex as they are. What we’re trying to do is give developers that context and visibility so they can proactively manage costs just like they do for performance.”

Currently, Harness’s Continuous Efficiency provides cost breakdowns by cloud provider, service, application, microservice, and environment, separating them out by Kubernetes cluster, namespace, workload, node, and pod. The tool shows costs for idle workloads and pinpoints changes made to code or infrastructure so that developers can troubleshoot when costs rose according to changes made. One of the key features of the tool, said Burton, is the fact that there is no tagging required.

“Everything you see is without tagging. Tagging is really the pain point. If you think of the analogy with continuous delivery, scripting is the pain point in CD, and tagging is the same with cost,” said Burton. “That’s really the only solution teams have today, and that’s simply adding metadata to all of the compute nodes manually. Obviously, in a cloud environment, that’s constantly changing.”

Feature image by Michael Longmire on Unsplash.

At this time, The New Stack does not allow comments directly on this website. We invite all readers who wish to discuss a story to visit us on Twitter or Facebook. We also welcome your news tips and feedback via email:

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.