Swim.ai Open Sources the Core of its Streaming Edge Compute Platform
The difference between computing on the edge and in the cloud, says Swim.ai co-founder and chief architect Chris Sachs, is far less about the location of compute, but rather the approach. Sachs says that the definition comes down to the way you deal with the data and when.
“The difference between the edge and the cloud has a lot to do with databases. Cloud applications, in our view, are database-centric apps,” said Sachs in an interview. “To us, the edge is not. By the edge, we don’t mean devices, although we do run on physical edge devices, and we don’t really mean a place at all. We mean apps driven by data as it flows, rather than post hoc after it’s stored in a database, whether it’s in a cloud or on a physical edge. We don’t care where we physically run.”
Swim.ai calls itself “the world’s first application platform for building stateful, real-time, distributed applications.” Thursday, it released the core of that platform as open source under the Apache 2.0 license. The platform provides developers with a single integrated stack that enables them to “develop applications that work with existing hardware and software without depending on a database, message broker, application server, or cloud storage,” according to a company statement.
Sachs says that SWIM’s edge compute platform contrasts the traditional cloud computing paradigm by processing real-time as it arrives, not after it is stored in a database. He divides applications into two general categories — database-centric and data-driven, with Swim enabling the latter.
“The stateless REST paradigm of the cloud is great for the cloud, but not awesome for the edge. When you depend on a database for everything, you’re restricted to using past data. The ultimate limitation is how much memory you have. Database-centric apps by their nature are very centralized,” said Sachs. “Data-driven is about analyzing what’s happening now as it’s happening. You’re limited not by memory but by time, by how quickly you can do your processing. If you have more data than you can store, you better process it quickly. These data-driven apps are very horizontally decentralized. It’s very difficult to move terabytes of data generated every day.”
Swim solves this by redesigning the software architecture itself, replacing REST with its “continuously consistent, multiplexed streaming protocol” WARP protocol, which it says creates the “the foundation of the world’s only streaming-first API.” Sachs explains that the WARP protocol makes it possible to handle the amount of data presented in edge scenarios. The company’s go-to example shows the real-time state of traffic lights, and Sachs explains that the data involved dwarfs that of traditional big data scenarios.
“CPUs operate on nanoscale time, while databases operate on milliseconds. The volumes of data are shockingly large when you’re dealing with streams of continuously generated edge data. It makes big data look like a drop in the bucket,” said Sachs. “Our traffic app in seven cities in the USA processes a half-million data points per second. Twitter is 3,000 tweets per second. The scale of the edge is staggering and the existing approaches to analyzing that data aren’t up to snuff for that problem.”
By contrast, Sachs argues that REST APIs can’t get near real time, whereas WARP operates at the latency of the network.
“Swim was created to solve the challenges of building real-time applications from multiple heterogeneous data streams,” said Sachs in a statement. “Stateless, REST-based architectures are quickly overwhelmed by high data volumes and storage costs. By using WARP protocol, Swim provides a highly efficient, stateful way to manage streaming data and to build applications that are continuously in-sync with the real world.”
Interested developers can visit developer.swim.ai and on GitHub. For the curious, visit ripple.swim.ai to see Swim in action.
Feature image via Pixabay.
