Stream Processing Could Replace ETL Deployments
In honor of this week’s Kafka Summit, we are resharing with a couple of insights based on a survey we conducted earlier this year about 800 IT professionals’ use of data stream processing. When asked specifically about Apache Kafka, 48% of respondents with streaming data use cases are using Kafka in production. However, Kafka is often used in conjunction with other stream processing technologies.
The survey found that 42% of stream processing users believe deployments are replacing existing technology within their organization. This does not take into account those that thing stream processing is just supplementing their existing stack.
Extract, Transform & Load (ETL) and messaging are the types of technologies most likely to see a replacement. Organizations that believe stream processing is replacing databases are more likely to use MySQL and Hadoop as data sources for stream processing. Neither of these technologies were designed to quickly handle the volume of data involved with streaming data use cases. Because these are open source data stores, people may believe it is easier to swap them out with another open source offering. In contrast, other data stores are less likely to be threatened by Kafka as they appear to be working well together. For example, 39% are using Cassandra with a stream processing use case, and of this group, 69% are using Kafka for stream processing.
