What news from AWS re:Invent last week will have the most impact on you?
Amazon Q, an AI chatbot for explaining how AWS works.
Super-fast S3 Express storage.
New Graviton 4 processor instances.
Emily Freeman leaving AWS.
I don't use AWS, so none of this will affect me.
Kubernetes / Open Source

Why Kubernetes and Kafka Are the Combo for DataOps Success

Lenses Co-founder Andrew Stevenson talks how Apache Kafka and K8s can increase the agility, efficiency and security of building real-time data applications.
Dec 8th, 2020 3:00pm by
Featued image for: Why Kubernetes and Kafka Are the Combo for DataOps Success

What is DataOps? Why is a real-time data platform essential to the use cases driving it? How can you build data pipelines with open source complexity?

In this episode of The New Stack Makers podcast — recorded live for KubeCon+CloudNativeCon North America, we talk to Andrew Stevenson, chief technical officer and co-founder of, about how Apache Kafka and Kubernetes can together dramatically increase the agility, efficiency and security of building real-time data applications.

“Data is a constant, and it’s moving around, no matter what technology we’re using,” Stevenson said.

Why Kubernetes and Kafka are the Combo for DataOps Success

So how do organizations make the most of that bigger-than-big data? Well, the majority of them have adopted open source tools Kafka stream processing platform or Kubernetes container orchestration system — or most likely both. But the complexity of these huge systems can be overwhelming to the point of ineffective. And with open source tooling, as Stevenson says, can come uncertainty.

He explained, “It’s a real struggle making sure the open source technology has the governance around it. It’s not that data is something new to people in organizations — they have data. But it’s the security aspect around it. And if you couple that with, quite often, the lack of visibility on open source technology — that is where it starts to fail.”

That’s where DataOps comes in, playing the role of management and self-service. It’s about creating a self-service pipeline and platform, like, which sits atop this open source technology, enabling all the data management including policies, governance, and internal knowledge and data stewardship, coupled with operations. Then, with all that in place, you can integrate with partners that can help you scale, all while compliant for data regulations like Europe’s GDPR. And, you can finally take your data management into production.

Everything is done in a controlled manner, within the context an organization’s data, so that the complexity of data no longer has to come at a cost.

Next sees its roadmap going towards answer key enterprise questions like:

  • How can I discover data?
  • How can I access it in a secure way?
  • How can I make it actionable?

And this can all occur within the context, governance and policies of an organization.

Group Created with Sketch.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.