The New Multicloud: Serverless, Application Flows and Automation
TriggerMesh sponsored this post.
When people hear the word serverless, it often evokes images of functions-as-a-service executed on Amazon Lambda. But that’s only one aspect. As the name implies, it is the abstraction of all the infrastructure needed to provide a cloud native application. This includes storage such as Google Cloud Storage, databases like Azure SQL, and analytics like those provided by Confluent’s hosted version of Apache Kafka. Notice my examples span three different cloud providers. That’s because as we move to a more serverless world, it’s likely that we will be taking a multicloud approach — where we consume the best of breed services from multiple providers. Not to be confused with the running of equivalent workloads on similar infrastructure on multiple cloud providers, a more dated definition and less interesting use case which cloud pundit and AWS watcher, Corey Quinn, lampoons in a recent snarky though well thought-out post.
I would agree that multicloud isn’t the norm today, but it was a vision laid out in the 1990s with Service-Oriented Architecture (SOA) — and it’s something we are a lot closer to in 2020.
Service-oriented architecture (SOA) is a style of software design where services are provided to the other components by application components, through a communication protocol over a network. A SOA service is a discrete unit of functionality that can be accessed remotely and acted upon and updated independently, such as retrieving a credit card statement online. SOA is also intended to be independent of vendors, products and technologies.
Wikipedia
The idea being that large software applications can be broken up as services that provide discrete functionality. These are “black box” services, where the underlying infrastructure is abstracted from the end-user. Services are combined to function coherently as an application.
Microservices and Event-Driven Architecture
As microservices have evolved as a design pattern, we see infrastructure providers like Google, Red Hat, Datadog, Twilio, Salesforce, and others providing both general infrastructure — like object storage — and more specialized services, like machine learning and monitoring. Ideally, in a cloud native world, you’d combine all these services using the best of breed approach to create an application.
So what is the framework for these services to interact? Events, Events, and Events. Events are simply messages from one system to another, where a change in one system is communicated to another system. For example, when a data set is uploaded to an object store, a machine learning service could provide some analysis of the content of the data set.
These events flow back and forth between systems, using the outcomes from one system to drive the outcomes of another system. This is an event-driven architecture (EDA). Streaming these events in real-time provides data pipelines, streaming analytics, and data integration for applications. Popular ways to share events are through streaming platforms like Apache Kafka or message brokers like Rabbit MQ (part of the VMware Tanzu portfolio).
The trend in EDA is being utilized by some of the biggest players in the industry. SAP for example is developing Kyma, a platform for extending applications with serverless functions and microservices based on Knative. It provides a selection of cloud-native projects glued together to simplify the creation and management of extensions. The Kyma project espouses the following vision around eventing:
- Enable customers to plug in messaging middleware that fits their requirements best, as well as to run multiple messaging implementations in parallel.
- Filter events and transfer only those with existing subscriptions (triggers).
- Support sending multiple events in a single request.
Cloud Run for Anthos, a serverless development platform on Google Kubernetes Engine (GKE), also powered by Knative, now includes an events feature that allows users to run event-driven systems on Google Cloud. Some examples include:
- Cloud Storage events trigger a data processing pipeline.
- BigQuery audit log events initiate a process each time a data load completes.
- A Cloud Scheduler event triggers a batch job, not unlike a cronjob on a Linux system but cloud-based.
The fact that these features are entering the product offerings indicates that the future is event-driven and is going to accelerate the automation of cloud native applications.
Application Flows
To this point, the vast majority of EDAs seem to gravitate around the more technical underpinning of software. For example, GitHub Actions, automates software workflows for CI/CD, build, testing and deployment. Another example is AWS EventBridge, which ingests events from SaaS like Zendesk and Auth0 and routes them to AWS services such as Lambda.
However, in the longer-term the ability to tie together cloud services that aren’t just related to what I’d term as general infrastructure — storage, compute, and networking — but rather, more specialized services, is what will provide more robust and powerful applications. For example, at TriggerMesh we are looking at ways to share data between Salesforce and data lakes, that can be further analyzed by Elastic. Or transforming and filtering events from GitHub to Twilio/SendGrid, or from Azure logs to Splunk, or from SonarSource static code analysis to Kaka — in each case exporting them as cloud events that can be consumed by services at Google, Microsoft, and Amazon clouds.
Summary
We are entering an exciting time. Infrastructure has never been so accessible from both a price and an ease-of-use standpoint. Microservices are a cloud native proxy for software libraries used in monoliths. When you combine them, you can make applications — not just client-server, but independent networked applications that span the cloud and even an on-premises data center. These applications are able to move faster in real-time, due to event-driven architectures and pervasive low latency networks. This is one of those rare times where the vision of a proceeding era has started to manifest itself as envisioned, and perhaps even exceeds the vision of the original authors.
Amazon Web Services and Confluent are sponsors of The New Stack.
Feature image via Pixabay.
