React in Real-Time with Event-Driven APIs
Where is my cab? What time will the train arrive? Was there a price drop in the stock that I invested in? Today, users increasingly demand ever more interactive experiences and they expect to be told when something has changed without hitting the refresh button.
So, how are you building apps that require (near) real-time updates based on remote events? Receiving notifications about someone liking your picture or reacting to your story on Instagram, or your Facebook news stream showing the latest updates about what your friends have been up to, are another couple of examples of frontend applications reacting to events in the backend. This is made possible through event-driven APIs, which are supported through various modes of asynchronous event-driven communication between the client application and the backend.
So, as developers and architects who create reactive web applications or expose APIs for the consumption of such applications, how exactly should we handle asynchronous or event-driven communication in the world of APIs where synchronous communication is predominant and most firewalls block non-HTTP traffic? It’s a matter of having an event-driven backend, choosing an asynchronous web API technology based on your exact async requirements, and a client that can work with remote events.
A grand majority of the APIs that make up the web today are synchronous APIs, where the client initiates all communication between the client and the backend by sending a request to the backend and waiting for a response. However, these request/reply API calls happen one at a time, in a pre-arranged sequence, and each interaction blocks the progress of the process until its completion. If the client application wishes to know about an update, it must continuously invoke the API for updates on a regular interval. This is known as polling and has been a common hack for client apps that need to become aware of new data. But, this method is largely inefficient and less reactive than if the backend immediately notified the client when an event occurs. The term “polling madness” was coined due to the fact that most of these polling calls are wasted because the data hasn’t changed.
When Should You Use Event-Driven APIs?
Event-enabling APIs is a relatively recent phenomenon compared to event-driven architecture (EDA), and it uses EDA to support scalable, real-time (or near-real-time), push-based communication in APIs published to third parties. Being event-driven is about taking advantage of events as they happen, and consumers can register their interest (subscribe) and react to events in real-time. Making APIs event-driven or asynchronous (along with an underlying server-side event-driven architecture) can eliminate the need for inefficient polling requests and send updates to the client (event subscriber) as soon as they occur. This provides a much better experience for users.
So, in a nutshell, event-enable APIs if:
- The application should push changes from the server to the client rather than waiting for a client request.
- The application must provide users with a highly active two-way communication channel.
- The application requires a large number of continuous interactions between the client and the backend that create scaling issues for the backend if synchronous APIs are used.
- The application needs to be able to monitor system-wide events.
Choosing an Asynchronous API Technology
Event-enabling APIs do come with multiple complexities — starting from what frameworks and networking protocols to choose, to building the reliability of delivery, and ensuring the scalability of a web-scale solution. The asynchronous protocols commonly used today solve the problem of real-time communication, but they solve different aspects of this problem in different ways — thereby making some protocols serve different purposes better than others.
For multiplexed, bidirectional streaming and for applications that need a considerable amount of messages from both ends of the connection, WebSockets is ideal. Server-Sent Events are especially useful in cases where a browser client is just interested in subscribing to a stream of data, without needing to communicate back with the server in the same connection. Webhooks can be used for a simple implementation of pushing notifications to one or a small number of servers. The newer async variants of GraphQL (Subscriptions and Live Queries) come with the benefits associated with GraphQL, however, the implementation is relatively complex and entails a considerable learning curve. gRPC and Kafka are generally used for communication between internal microservices.
What the Backend Architecture Should Look Like
While the client architecture must subscribe to state change events that originate from the backend, the typical backend architecture can be extended to create remote event connections with the clients via:
- Event-driven APIs (ideally exposed through an API management platform that supports eventing semantics).
- A message broker (if one is not already present).
- Microservices that publish and process state change events.
Although event-driven APIs do not explicitly require a broker, using an intermediary between event producers and consumers helps to implement the required patterns to deliver more manageable and scalable solutions. The broker receives events from IoT devices, change data capture (CDC) tools, other backend systems and services, and from client applications if two-way communication is enabled. It can then alert the services that subscribe to those events. A scalable microservices architecture (MSA) is the optimal architecture for complex event-driven backend services. These event-driven microservices can act as event subscribers or publishers in order to process events, handle errors, and persist event-driven states.
The remote event connection between the backend and web clients can be established through event-driven APIs powered by various asynchronous API technologies — such as Webhooks, WebSockets, Server-Sent Events, GraphQL subscriptions, etc. The backend uses these APIs to send events pertinent to the clients and also receive events that originate from clients.
Managing Event-Driven APIs
An event-driven backbone will manage the overall real-time data flow securely and at scale, while asynchronous APIs can be managed for external and internal consumption with a traditional API management solution that comes with inherent or extended capabilities to support event-driven semantics. Most organizations have basic event processing infrastructure, but many don’t have the capabilities to design, develop, test and manage event-centric APIs. Essential API management capabilities — particularly governance, access control, monitoring, analytics, and monetization — provided by API management solutions can be used to manage asynchronous and REST APIs. Some API management solutions already support Webhooks or/and Websockets.
Support for asynchronous APIs in API management tools gained traction with the AsyncAPI initiative standardizing event-driven APIs. AsyncAPI is an open specification format similar to the Open API Specification, but focuses on asynchronous interfaces. AsyncAPI is protocol-agnostic and lets you document how events are structured, their semantics, how consumers can subscribe, how events are delivered by a producer or consumer, and how endpoints are secured and authenticated.
Figure 1: A high-level backend architecture to support event-driven APIs alongside synchronous APIs.
Conclusion
Synchronous APIs are ideal for many scenarios — particularly if you need an instant response. In other cases, when clients need to be informed of events or the processing required for the response happens at a different time, ordinary synchronous messaging becomes tricky and event-driven APIs can aptly address such needs.
EDA is flexible enough that you can start with simple notifications, or combine two-way event communication with a synchronous approach to deliver an optimal architecture. So, using an event-driven design alongside a traditional request/response design for your APIs brings the best of both worlds to build highly scalable and reactive web applications. Furthermore, combining traditional API management capabilities with an event-driven architecture provides tremendous value additions to expand business reach and adoption.
For more detailed information about creating event-enabled API architectures with asynchronous API technologies and API management platforms read this whitepaper by the author.