Confluent Brings Visual Design to Kafka-Based Data Streams
Data platform provider Confluent has unveiled a visual interface, called Stream Designer, that enables developers to build and deploy streaming data pipelines on the company’s Kafka-based platform and cloud service.
“With Stream Designer we want to democratize this movement towards data streaming and make real time the default for all data flow in an organization,” said Confluent CEO Jay Kreps said, in a press release.
The company took the wraps off the new user interface at its Current 2022 Summit, which was held this week in Austin, Texas. The New Stack sat down with Confluent’s Principal Product Manager Jon Fancey to discuss the product.
Stream Designer provides a SQL-based approach to build data flow logic.
In short, Stream Designer aims to abstract away all the nuts and bolts of building Kafka data streaming pipelines without abstracting away so much that there’s no visual pinpoint between the source code and the UI.
Fancey expressed the importance of providing a shorter “on-ramp” for developers implementing Kafka into their applications. The team who built this product spent much time considering all the different use cases, he said. Even experienced Kafka developers may find the interface helpful.
Tell Us More!
Stream Designer doesn’t add any new abstractions and follows a 1:1 mapping pattern with the usual Kafka build practices. Stream Designer will create the topic on your behalf and add it to the canvas in the graphical interface. It will also add it to the source connector once the topic is specified. It builds out all the logic needed for the data to flow from producer to topic to consumer.
In addition to the ease of pipeline creation, Stream Designer also offers a visual representation of the data flow from connection to source into the topic. In-pipeline message inspections are also available.
Custom data pattern capabilities are also available., such as for sales data or data that needed to be broken up by regional location. Once you “start to build and understand your first pipeline, you can extend and enhance that in whatever ways [you] want,” Fancey said.
Stream Designer doesn’t have a runtime so this will not be the tech that bottlenecks the application. Stream Designer crafts the artifacts that manifest as a pipeline. Once the pipeline is built and code is generated, the code gets deployed either through the UX or the CLI.
Detailed metrics and schemas are automatically registered in Confluent’s Schema Registry, pipeline iteration, and multi-user environments are also allowed.
Code to Interface — And Back Again
When asked what the largest technical challenge the Stream Designer team faced was, Fancey didn’t hesitate: The. visual. interface. But not just the part of the interface that developers, or anyone needing to see the GUI, could see. It was the whole idea behind it. What was important? What could be abstracted away? Where was the line between necessary and nuisance?
In the end, it all came down to, where it usually does, in the details.
“The details matter. It was one of our principles from the start that we had a clean mapping, because we’re aiming at developers and allowing them to be being able to go backward and forward from the code to the graphical canvas,” Fancey explained the challenge.
If too much source code is removed then the canvas, “becomes quite detached from what you’re wrapping,” and because this was built essentially for those who do understand source code and can do the hard learning if needed, “it’s important to understand how this thing works.”
The end result is a pipeline and a canvas that complement one another well. The nodes in the graph on the canvas reflect the pipeline automatically. There are relationships between the nodes and connectors so the developers working on the application can go behind the scenes and see which topics were created by Stream Designer. Because while there is a very detailed canvas, the source code is also visible, so much so that devs can switch from screen to screen between the two.
Only time will tell how this new product is incorporated by developers but the Confluent Stream Designer team will not be disbanded anytime soon. General Release is just the beginning and there are plans for additional features and larger platform integrations such as an integration with Stream Lineage, though there is no release date for this feature.