What news from AWS re:Invent last week will have the most impact on you?
Amazon Q, an AI chatbot for explaining how AWS works.
Super-fast S3 Express storage.
New Graviton 4 processor instances.
Emily Freeman leaving AWS.
I don't use AWS, so none of this will affect me.
Data / Operations

How Apache Flink Delivers for Deliveroo

A pair of software engineers from Deliveroo share details about the company's transition to Apache Flink to improve real-time data streaming.
Sep 20th, 2023 6:54am by
Featued image for: How Apache Flink Delivers for Deliveroo

Deliveroo is a leading food delivery company that builds tools and services using Apache Flink.

The Flink framework offers a distributed processing engine for “stateful computations over unbounded and bounded data streams.”

Deliveroo has a three-sided marketplace. They have delivery drivers, restaurants, and customers who place orders on the application to get food or groceries delivered to their door. Joining us to discuss using Apache Flink and the Amazon Managed Service for Apache Flink were two engineers from Deliveroo: Felix Angell and Duc Anh Khu.

Deliveroo sought to do more real-time streaming. They explored behaviors to understand the customer journey and how those customers use the Deliveroo application.

That meant modernizing to a more stable platform. The old platform could scale up but not down due to earlier decisions made about the technology. They looked at other services such as Apache Spark and Kafka Streams. But Flink had feature parity with the legacy platform, and large companies used Flink for production.

Deliveroo started experimenting with Flink as a proof-of-concept on Kubernetes. They used third-party operators but found many needed more support and were not maintained. They turned to the Amazon Managed Service for Flink (MSF), which allowed the Deliveroo team to focus on its core responsibilities, such as CI/CD and taking updates to production.

Angell said he would like more Deliveroo teams using Apache Flink. Duc said they move very fast to roll out the latest product. And that means their modeling could be more flexible to adapt to that demand.

Flexibility comes with a cost, he said. Sometimes, you need to remodel things, though other times, you need to normalize the data model. It would help make that process easier for teams to do.

“And for me, one of the features that we would like to see is a self-serve configuration from MSF, so that we can just tweak some of the low-level configuration as well as auto-scaling requirements based on application metrics,” Duc said.

Learn more about Apache Flink and AWS:

Kinesis, Kafka and Amazon Managed Service for Apache Flink

Apache Flink for Real Time Data Analysis

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Pragma, Real.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.