The Need for Event-Driven Integration
Developer sentiment is clear: developers’ and architects’ interest and adoption of event-driven integration is a mainstream objective, as confirmed by a research project on the current state of API integrations:
Most of these organizations are ambling to create their own “Composable Enterprise” and will rely on multiple cloud-based services to deliver this vision. Yet many of these SaaS applications are islands that haven’t been optimized for integration, and many developers depend on polling — lots and lots of polling — to create the workflows and integrated scenarios they need.
The term “Composable Enterprise” was first coined back in 2013, by Jonathan Murray, then executive vice president and chief technology officer of Warner Music Group. Murray effectively defined a radical new operating model for businesses as they grapple with demand for new digital products, continuous innovation and increased agility. The original thesis can be found here.
For many application providers, there is a long way to go to offer the features and event-driven capabilities developers expect, so it’s important to make progress now. Companies should stop polling and move to event-driven integration models that scale.
Real-time Demands Are Driving Event-Driven Requirements
The need for, and increased interest in, event-driven architecture is arguably attached to the demand for real-time application integration and data movement. The world of real-time is perhaps misunderstood, used in the wrong context, or at best an overloaded term where the tolerances for delay or latency vary based on the use case.
What’s certain, is that many hold strong opinions on the definition, but in this case real-time implies web technologies that enable applications to react to relevant business events as they happen — or at least within an understood period of latency (a 4ms delay may render a financial trading platform uncompetitive, while 400ms delay is easily tolerable for most application integration use cases).
Across much of today’s application integration, there is no ability for systems to react to events in real-time. Typically, system A (the Client) will ask system B (the Server) if new data is available based on a set of query parameters — a well-understood request/response pattern native to the web. An event-driven pattern is the foundation of real-time application integration — removing the need for clients to ask repeatedly if a given system has new data available.
An event represents something that happened within a system, a user action or even an action caused by existing request/response interfaces. Each of these events will typically involve new or changing data that described what just happened. For event-driven integration, there is no need to ask a given system if an event happened. Instead, the system will tell any other interested system or application about it. The event and associated data will then be delivered to other systems, allowing them to react in real-time.
Event-Driven Technology
The good news is, to enable real-time there are a variety of technology options and architectural best practices to embrace. The WebSockets standard is now well established as an efficient mechanism to stream data in real-time, and WebHooks is gaining momentum as the protocol and pattern of choice for event-driven APIs. The arrival of HTTP 2.0 and Server-Sent Events represent an ever-evolving landscape of technologies that may be used in other event-driven use cases.
- Server-Sent Events: First-off, for many the term Server-Sent Events (SSEs) may be unfamiliar. For the past few years the specification has been in a state of flux and has definitely been overshadowed by the more fully featured communication protocol (WebSockets). However, the idea behind SSEs should be familiar: allowing a Web application to “subscribe” to data updates originated (sent) by a server, and subsequently be notified as these data events occur. The problem here is that while it removes the polling technically, it also removes much of the control from the user, and forces them into a passive state.
- WebSockets: WebSockets provide a richer protocol to perform bi-directional, full-duplex communication. Having a two-way channel is more attractive for things like games, messaging apps, interactive experiences and for cases where you need real-time updates in both directions. But here again, the solution isn’t perfect since the integration pattern effectively breaks the semantics of HTTP.
Over the past 18-24 months, there has been increased chatter about the limitations of Web APIs, and how architectural choices must be made on merit versus conceptual elegance. This has led to the evolution of standards, such as v3.0 of Open API Specification, which now includes a way to document WebHooks as part of a RESTful API.
WebHooks work by automatically posting new event data to a user-defined URL which is monitored by the user’s linked applications. Each time a new event is posted to the URL, the linked applications update to include the new data. Unlike polling, which delivers usable data on less than two percent of requests, WebHooks only update when new information is available. Because of the increased efficiency, over 82 percent of developers surveyed by Wufoo indicated that they would rather receive new data via WebHooks instead of with polling. Despite this preference, only 29 percent of APIs currently support WebHooks.
There is no single design decision that perfectly answers every integration question, but the perils of polling are clear — contributing to network overhead, cost and poor user experience. Consider alternatives like WebHooks when you’re building your next Web API or integration scenario.