Build Next-Generation Cloud Native Applications with the SMOKE Stack
TriggerMesh sponsored this post.
Enterprise technology needs to help organizations take action in real time. Doing this effectively means modernizing application architecture from batch processing to event-driven.
Serverless computing is an event-driven architecture that abstracts infrastructure, so developers can focus on writing the application code. With serverless, application teams don’t need to worry about the complexity of maintaining, patching, supporting and paying for infrastructure that they need on an elastic basis. This makes serverless perfect as the glue to integrate services from anywhere.
At TriggerMesh, we think serverless is only the beginning. The real power comes from what serverless enables. Serverless architectures allow even the largest enterprises with years or decades of legacy code to break out of the constraints of their own data centers and a single cloud. Open source, standards and specifications free enterprise developers to mash-up services from on-premises and any cloud, to rapidly compose event-driven applications that support high velocity — so that you can bring new features and products to market fast.
For enterprises to succeed in a hybrid world that moves at breakneck speed, they need platforms that harness the power of the following five trends:
- Serviceful
- Mashable
- Open
- Kubernetes
- Event-driven
I call this combination the SMOKE stack and it will power the next generation of cloud native applications. The SMOKE stack gives enterprise developers the flexibility to respond fast to changing markets and to make the best choices about on-premises, hybrid, or multicloud.
Serviceful
Serverless and functions-as-a-service(FaaS) are often conflated, but serverless is much more than FaaS. That’s why my colleague and cloud-native expert, Sebastien Goaguen likes to say serviceful, not just serverless. In a cloud native world, applications are a combination of services — think of these services as very similar to libraries in monolithic applications. Because serverless abstracts the complexities of infrastructure, it allows developers to focus on applications comprised of event-driven functions that respond to a variety of triggers.
A FaaS platform (such as Amazon Lambda or Google Cloud Run) lets users write small pieces of code that are executed based on a change in another system. A FaaS platform takes care of trigger-to-function logic, passing information from one function to another function, auto-provisioning of container and runtime (when, where, and what), auto-scaling, identity management, and more. It also transparently handles provisioning and security.
The cost savings, scalability and reduction in operational complexity from serverless are real. Digital publisher Bustle experienced approximately 84% cost savings by moving to a serverless architecture, Thomson Reuters processes 4,000 requests per second with serverless, Major League Baseball Advanced Media ingests, analyzes, and stores over 17 petabytes of data a second. And these are just a few of the more notable highlights.
Mashable
LEGO bricks are a common metaphor for building cloud native applications. The big building blocks are libraries and cloud services that developers work with — such as machine learning, analytics platforms, transaction platforms, conversational tools, etc. These building blocks allow developers to rapidly construct cloud native applications that provide tremendous business value and that can be updated continuously.
This is all very straight forward when all the building blocks you need are from a single cloud. But increasingly, that’s not the way developers work — and it’s not going to work for enterprises with use cases that require substantial on-premises deployments. These developers will inevitably need to use a block that either lives in another cloud or on-premises, as part of their cloud native application.
In order for developers to build bridges across these deployment chasms, they need a way to define an event flow that is agnostic to where the service is running.
Open
Solving the problem of cloud native integration depends on open specifications and open source. One important project is the Knative project, which provides serverless computing (including Knative eventing) on top of Kubernetes. Knative is supported by Google, IBM (including Red Hat), SAP, and VMware (including Pivotal) and is an important building block for automated event-driven infrastructure. Open not only means open source but open specifications like CloudEvents from the CNCF, which provides important industry standards for communication across cloud services.
Open source is everywhere, and enterprises are sold on it. In Red Hat’s The State of Enterprise Open Source 2020 Report, 95% of respondents said that open source is strategically important. After Security, Cloud Management Tools are the next most popular use for open source identified by IT leaders surveyed by Red Hat.
Source: Red Hat
In the past, companies could build a business around a strong open source core product, without a large platform company introducing the same product as a service. Today, however, the latter is the norm.
Competitors have always been contractually allowed to offer another company’s OSS product as a service. We’re witnessing the rise of highly-integrated providers taking advantage of their unique market position to offer “as-a-service” versions of OSS products, and offer a superior user experience as a consequence of their integrations. We’ve seen it happen with Amazon’s forked version of ElasticSearch, which Salil Deshpande neatly described in TechCrunch as both “self-interested and rational.”
To respond to this breed of competitor, companies like Cockroach Labs adopted an extremely permissive version of the Business Source License (BSL). CockroachDB users can scale it to any number of nodes. They can use CockroachDB or embed it in their applications (whether they ship those applications to customers, or run them as a service). They can even run it as a service internally. The one and only thing that you cannot do is offer a commercial version of CockroachDB as a service, without buying a license. Other companies like Confluent and Elastic have done the same.
While these licenses don’t fall strictly under the Open Source Definition, they provide most of the same benefits.
Kubernetes
In your push to satisfy customers’ unpredictable demands, microservices architectures are perhaps your number one ally. Microservices are independently packaged and deployed via containers. As microservices applications grow, orchestrating the containers and their dynamic access to compute, storage and networking can become complex. The leading solution to handle this complexity is Kubernetes, which is supported by every major cloud and infrastructure vendor. Kubernetes is fast becoming the operating system for the cloud.
Kubernetes is an open source container orchestration system for automating deployment, scaling and management of containerized applications. It was originally designed by Google and released as open source in 2014, and is now maintained by the Cloud Native Computing Foundation.
In the last 20 years, few technologies have seen a more meteoric rise than Kubernetes. The container orchestration platform was initially released in 2014, and by 2019 was in use by more than 86% of organizations. Part of the appeal is that users can run Kubernetes in the cloud as well as self-managing it on-premises, paving the way for an open hybrid cloud. Surveys report that more organizations self-manage Kubernetes, 44% percent, than any other deployment model. Among managed Kubernetes offerings, Amazon EKS leads with 27%, followed by Azure AKS (16%), Google GKE (12%) and IBM Red Hat OpenShift (12%).
Knative
Knative is an open source Kubernetes-based serverless platform that provides a set of middleware components to build modern, source-centric, and container-based applications that can run anywhere: on-premises, in the cloud, or even in a third-party data center.
In his talk at NDC London 2020, Google Developer Advocate Mete Atamel describes Knative as helping to resolve a previous tradeoff that developers had to make between serverless or containers. With Knative, you get both: the flexibility and familiarity of containers, along with the zero-touch provisioning and fast iteration of serverless.
Knative does two things: serving and eventing.
- Serving builds on Kubernetes to support the deployment and serving of serverless applications and functions. Serving provides the autoscaling — including scale to zero — feature of FaaS, as well as fine-grained traffic control using modern network gateways.
- Eventing provides building blocks for consuming and producing events that adhere to the CloudEvents specification developed by the CNCF Serverless Working Group. It includes abstractions from event sources, and decoupled delivery through messaging channels backed by pluggable pub/sub-broker services.
Both Google Cloud Run and Red Hat OpenShift Serverless are based on Knative. Atamel shared a very relatable use case of a Knative event-driven application to query and report on COVID-19 cases from specific countries once a day. This application, which runs on GKE, uses the Google CloudScheduler service, connects to the COVID-19 public dataset program on BigQuery, uses Matplotlib to create the visualizations, and sends emails with the SendGrid free tier. The illustration below, from Mete’s blog, shows the architecture.
Knative Event-driven Application to Report on COVID-19 Cases. Source: Mete Atamel, who provides the instructions on how to build the pipeline and the code and configuration in his Knative Tutorial.
With the amount of investment pouring into serverless from Google, Red Hat, AWS, Oracle, and others, keeping your organization in a position to be able to take advantage of new cloud capabilities is vital.
Event-Driven
Think of event-driven architecture (EDA) as enterprise service bus meets microservices architecture.
Confluent CEO Jay Krebs wrote that applications increasingly interact directly with other applications in a mesh, before returning some output to a person. The currency for these application-to-application interactions is the event.
Monolithic applications simply cannot keep up with businesses’ need to rapidly bring new features and services to market. Microservices represented the first step to provide a quicker way to develop and deliver business-aligned services. As the number of these services grows, the next step is to integrate them through event notifications. And this is motivating enterprises to embrace EDAs.
EDAs are composed of loosely-coupled, distributed services connected by asynchronous events. These systems are perfect for today’s uncertain times, because new services and data sources can be added or removed on the fly without disrupting the existing application flows.
It is becoming evident that EDAs are the best way to take advantage of the explosion of new cloud services and the exponential growth in data. The opportunities for businesses to leverage data and services to build apps that deliver better experiences is only limited by the flexibility of their systems.
The rub is that new research finds a massive 93% of companies operate in a multicloud environment, which can make integrating event sources and consumers from different clouds and on-premises a big challenge.
Some of the more common EDA use cases include stream processing, data integration, and customer journey mapping. If we look at data integration, you may have a primary product database on-premises — maybe in an OracleDB. But perhaps you’ve migrated ERP to the cloud, and you run Salesforce. An event-driven application flow to integrate these systems would benefit the entire organization, so that whenever there is a new price in the product database, this triggers an event that is consumed by the ERP systems for forecasting and by sales for RFPs and quotes.
In the same Confluent blog, Kreps predicts “The purpose of an application, in this emerging world, is much less likely to be serving a UI to aid a human in carrying out the activity of the business and far more likely to be triggering actions or reacting to other pieces of software to carry out the business directly.”
Perhaps this is the most pragmatic way for companies to continue leveraging high-investment, on-premises systems, while also accelerating the move to cloud for digital transformation.
Conclusion
Armed with the powerful technologies in the SMOKE stack, enterprises of every stripe can modernize their applications to meet whatever challenge or opportunity 2020 brings us next. If the COVID-19 curveball has taught us anything, it is that our ability to respond rapidly to dramatic and sudden change has never been more important. The SMOKE stack, we believe, represents the best answer to the question on the mind of every architect: how can we be ready for anything? For more, grab the new SMOKE Stack Whitepaper here.
Amazon Web Services, the Cloud Native Computing Foundation, Confluent, Red Hat, VMware and Cockroach Labs are sponsors of The New Stack.
Feature image via Pixabay.
