Concern About ‘State’ Lessens as More Applications Use Stream Processing

Persisting data in a microservices architecture becomes less of an issue as users gain more experience with containers, microservices and modern databases, according to a survey about stream processing conducted by The New Stack in partnership with Lightbend.

Let’s be clear, 68% of respondents said handling state is at least to some extent an obstacle to deploying more applications within microservices architectures. This is not a permanent barrier though, and many of the issues are mitigated once developers actually get experience with newer technology.

Only 18% of respondents believe state is not at all an obstacle to microservices adoption. However, among those using streaming in more than half of their applications, 41% say state is not at all an obstacle. This is because advanced users of stream processing are using design patterns and databases that are better able to handle data in movement. In other words, new applications designed so that they can utilize both batch and real-time data processing are less likely to have problems with state management.

Many concerns about handling persistent storage are due to existing applications being designed for older databases and architecture. Users of modern data stores like Cassandra, MongoDB and Redis are less likely to believe state is inhibiting adoption of microservices. Since 71% of respondents were extremely or somewhat comfortable with refactoring an application for microservices, connecting new stream processing use cases to these cloud-native databases should not be a problem.

In 2018, we reported that usage of Kafka to preserve and communicate state microservices architectures had risen dramatically, at least according to a survey of Kafka users. In 2019, we just narrowed the focus of the question so that Kafka’s core storage and messaging capabilities were distinguishable from its streaming engine. Using this approach, we found that early adopters of some stream processing technologies are more likely to be troubled by state. Among those that think state is greatly inhibiting further microservices adoption, production-level adoption of Apache Kafka, Apache Spark Streaming and Elasticsearch was on average 36% higher than the sample as a whole.

People who did not use the right tools to address the right problems are a reason why persisting state continues to be cited as a concern. However, these concerns appear to abate as the adoption of new design patterns associated with stream processing and modern databases take hold.

More Data from the Report

• Forty-two percent of stream processing users believe deployments are replacing existing technology within their organization. This does not take into account those that thing stream processing is just supplementing their existing stack. Organizations that believe stream processing is replacing databases are more likely to use MySQL and Hadoop as data sources for stream processing. Neither of these technologies were designed to quickly handle the volume of data involved with streaming data use cases. Since these are open source data stores, people may believe it is easier to swap them out with another open source offering.
• Based on additional survey questions, we found that organizations with a high percentage of applications using stream processing are utilizing more persistent datasets and storage models. This is consistent with the fact that they are less concerned about persisting state. For much more detail and charts, download the full report.

Lightbend is a sponsor of The New Stack.

Lawrence has generated actionable insights and reports about enterprise IT B2B markets and technology policy issues for almost 25 years. He regularly works with clients to develop and analyze studies about open source ecosystems. In addition to his consulting work,...

Read more from Lawrence E Hecht