A New Day for Streaming Data with Kafka and Confluent
Launched by the creators of the Apache Kafka distributed streaming platform, Confluent not only maintains Kafka, but is on a mission to teach the world what a data-driven application ecosystem can do for business. On today’s episode of The New Stack Makers, Confluent CEO and co-founder Jay Kreps was interviewed by TNS associate podcast producer Kiran Oliver to discuss how Kafka has evolved over the years, what businesses just getting started with data streaming can expect from the platform, and how developers can tackle some of the challenges commonly faced when working with constant streams of data.
A New Day for Streaming Data, with Kafka and Confluent
“It was almost an unknown area to talk about streaming anything. When we released Kafka as open source originally, even then I don’t think people knew what to make of it,” said Kreps.
Part of what companies just getting started with Kafka should consider is how they’re getting their data from Point A to Point B. This is an area where Kafka shines. In the past, data was usually processed once a day in large batches, similar to banks processing transactions on a daily basis.
“At a high level, what Kafka is doing is allowing you to process data as kind of a stream. You can think of it as a flowing river of data. The result of cloud or on-premise migration is their data is spread across all of that. Getting that data where it is supposed to be is the first use case. Streaming that data to where it is supposed to be,” said Kreps. Creating applications that not only process and ingest data, but react to it and adapt is another thing even small businesses can consider when getting started with data streaming and processing. Kreps highlights a use case where retail giant WalMart has put Confluent to use in its own inventory system, allowing it to keep better track of stock levels and ensure store-level item counts are accurate.
In today’s modern world, if for example, a traveler was playing a game on their phone with others, then lost connection for the flight, what would happen to their now potentially hours-old data? “The only time that matters is when the data occurred in real time on your phone, not when it arrived on the system,” Kreps noted, adding that, “Another problem you have to solve is what do you do when you change your code? What you do is you basically just rewind time, then fast forward and process really quickly off that same stream. By doing that, you can use the same system and exactly the same code. That idea of being able to rewind and replay time is the way things are going to end up just because it’s so much simpler.”
Ultimately, Kreps hopes that data streaming will bring about the future of application development and data processing. “Our goal is to make this the default way to build applications. I think we’re just getting to that point now, where it’s easier to build apps this way. It’s not just faster, it’s also easier.”
Feature image via Pixabay.