Memphis Cloud Focuses on Making Event Streams Easier to Manage
The market for real-time data platforms continues to expand, with more alternatives to Kafka popping up. One of the latest packages, Memphis.dev is a message service broker designed to make life easier for the administrator, noted Yaniv Ben Hemo, CEO and co-founder of Memphis, the company behind the technology.
This month, Memphis launched a real-time streaming data platform, Memphis Cloud, based on the open source Memphis.dev.
Once, real-time data operations was confined into the limited provinces of financial companies and military contractors, both of whom have strict temporal requirements for their systems.
These days, however, more enterprises are heading down that path, adopting microservices and event-driven architectures that require real-time large-scale messaging across components. Also, Ben Hemo explained, many companies prefer a pay-as-you-go option for computing services, especially those companies with highly varying traffic loads such as GrubHub or Uber. This approach also leads down the path of real-time messaging.
Potential uses of a real-time platform may not require 100 messages a day, Ben Hemo said. Even a system with only a million messages a month would still benefit from a solution such as Memphis. “They still need a method,” he said.
Difficulties of Real-Time Messaging
In this ecosystem, Memphis.dev co-founders all knew what a headache messaging brokering was to manage. Data got lost, poor observability, and scalability was difficult.
Another major problem was implementation costs. The founders did some research and found that even very large organizations found message brokers to be a major drain of resources and talent that could be better used elsewhere. This was true with the managed versions of Kafka and Pulsar, with as well as with other approaches such as Rabbit MQ, and commercial services such as Amazon MKS and SQS, Ben Hemo said.
One of the biggest issues in this space has been management of real-time systems — the challenges of setting up and managing clusters.
For instance, Kafka involves stitching together a number of components, which may include Kafka Streams, Schema Registry, another tool for data transformation and Apache Flink. And in most cases, deployment requires craftiness in either Java or Kubernetes.
“It’s a nightmare,” Hemo said.
As a service, Memphis aims to make all this easier to manage. The company likes to tout the service not just as a message broker but as an “intelligent event processing platform,” Hemo said. Thus far, the open source version, based on the Cloud Native Computing Foundation‘s NATS protocol, has been used in over 5,000 deployments.
Hemo noted that Redpanda was written in largely in C++ to prove performance time (Kafka was written in Java, while NATS was built with Golang). The users the Memphis founders talked with weren’t particularly interested in improving performance but rather wanted smoother ops.
Memphis also works very well with Kafka, allowing users to extend out their existing Kafka deployments with new functionality, Hemp said.
With Memphis, stream processing can be done in 15 different languages. It runs on Kubernetes and can be used on-prem, a cloud or across multiple clouds. Out-of-the-box capabilities include monitoring, high availability, and self-healing. It is the first platform to provide real-time schema compliance, relieving the client developer from writing code to normalize the data as it arrives at the end-point.
To ease troubleshooting, every error message is written into the dead letter queue so the failure can be reconstructed by the developer. Developers can be messaged by Slack or phone message and they can build self-healing recipes that can automatically kick in until help arrives, if it is still needed.
Future development efforts will center around GitOps, automation enablement, multitenancy, partitions, and read replicas, and reconstructing APIs in a modular, open format, according to the company.
Memphis.dev was founded in 2022. It is privately held and has secured $5.58 million in seed funding co-led by Angular Ventures and boldstart ventures.
