Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
Cloud Native Ecosystem / Cloud Services / Open Source / Security

DevOps, Pandemics and 2022’s Cloud-y Future

Jan 18th, 2022 6:29am by
Featued image for: DevOps, Pandemics and 2022’s Cloud-y Future
Feature image via Pixabay  

When I wrote the post “DevOps, DevApps, and the Death of Infrastructure” last year, at the time I saw many things coming, often influenced by the pandemic. In 2020, McKinsey and Co. found that digital transformation accelerated globally due to the pandemic. As we finished 2021 with new considerations for continued digital transformation, it is no longer a trend but a business requirement for companies that want to stay relevant in a socially distanced world.

DevOps and Cloud Computing Trends for 2021 

Mark Hinkle
Mark has a long history in emerging technologies and open source. Before co-founding TriggerMesh, he was the executive director of the Node.js Foundation and an executive at Citrix, and Zenoss, where he led its open source efforts.

I credit my career at the forefront of many computing trends for my knack for seeing what’s coming next. I worked at the first national ISP, at a Linux desktop company during the year of the Linux desktop (yes, 1999 – now).

I drove through a blizzard in Belgium with Patrick Debois, the father of DevOps, before DevOps entered the enterprise software lexicon. I was an executive at one of the first cloud computing companies, and now I am fortunate to work at a cloud native company, TriggerMesh.

Here’s a list of things I believe will drive enterprise software and cloud computing in 2022.

From Composable Infrastructure to Composable Applications

Composable infrastructure allows for the abstraction of computing, storage and network resources to be controlled via an API. As a result, composable infrastructure makes data center resources as readily available as cloud services so they can support private and hybrid cloud solutions. Today, we have the building blocks to construct massively scalable businesses on this infrastructure, opening the door for an even more powerful evolution of composability.

As infrastructure abstraction grows, we are beginning to see applications composed of cloud services to create cloud native services. Soon, we won’t compile software but rather chain microservices together into cloud native applications, something I call “DevApps.”

For example, we have seen the growth of HashiCorp’s Terraform to deploy cloud infrastructure, the latest evolution of infrastructure as code. The next logical step is what we at TriggerMesh call integration as code. The same DevOps practices used to manage infrastructure deployment should be used for integrations. To fuel this next era, TriggerMesh open sourced our cloud native integration platform to help enterprises build composable cloud native applications. This composability lends itself to automation for deployment and interaction of services, as well as the addition of artificial intelligence and machine learning to create more innovative applications in future years.


In the past, security has often been an afterthought, something added at the end of the project, not a design point at the beginning. Nowadays, development cycles last days or weeks or at most months. To be effective in observing DevOps practices, you need to rapidly develop new features and deploy them frequently without sacrificing security. This is generally a good practice, but according to Sonatype’s “2021 State of the Software Supply Chain” report:

“In 2021, the world witnessed a 650% increase in software supply chain attacks aimed at exploiting weaknesses in upstream open source ecosystems. For perspective, the same statistic was 430% in the 2020 version of the report.”

And when the Log4j exploit hit in December, the developer community got another rude wake-up call around security. Safety never takes a vacation, and neither should security. In response to a growing number of threats, we will see an increase in technologies providing supply chain security, including startups like Chainguard, founded by Google ex-pats to ensure that build systems are secured before the software is deployed into production.

Event-Driven Architecture (EDA)

There has been a ton of fanfare around serverless for the past few years. However, I believe the hype isn’t just because of the serverless success stories but for the overarching category of the event-driven architecture of which serverless is a part. Serverless is EDA. An event triggers a serverless function that scales up and then down as needed, for example, when EventBridge consumes events from AWS Kinesis then triggers a Lambda function. Though the adoption of serverless has been good, the adoption of the foundational technologies, like event-streaming technologies Apache Kafka and newcomers like Apache Pulsar, is growing and providing the foundation for even more interesting examples of EDA.

A report by Coleman Parkes sponsored by Solace found that 72% of global organizations take advantage of event-driven architectures. These organizations saw many events resulting from adopting EDA and are seeing the benefits of real-time data and automation.

Event-driven architecture will increase organizations’ ability to access data faster in real-time and take action on that data through automation and data synchronization.

Private Cloud (Native) 

In the early days of cloud computing, there was a lot of talk about public and private clouds. Today, the word is about cloud without the qualification, with the adoption of Kubernetes growing fast, with production use increasing to 83% in 2020. Not only was Kubernetes adoption growing, but on-premises deployments of Kubernetes were rising at a much faster rate than managed cloud offerings.

And an increase in private cloud usage was also on the rise. So this is why I think the trend is not just in cloud computing but just cloud native.

Open Source First

Twenty years ago, many enterprises were wary of open source software. Today, we see open source as a standard for the cloud infrastructure. Linux, KVM, and Kubernetes provide the foundation for many cloud services. Tooling from HashiCorp, Elastic, Confluent, Kong and numerous other projects is providing the backbone for managing cloud infrastructure. This trend is only growing. I suspect projects coming from the CNCF Sandbox will continue to mature and become part of that production ecosystem along with many others.

According to Red Hat’s “State of Enterprise Open Source” report, 79% of enterprises expect to use enterprise open source software for emerging technologies to increase over the next two years. Also, 87% see enterprise open source as more secure than proprietary software.

Today, the tables have turned, and even the most conservative industries take an open source first stance when sourcing their infrastructure software. Here are some open source projects related to cloud computing and DevOps for 2022.

  1. Apache Pulsar — Apache Pulsar is a cloud native, distributed messaging and streaming platform created at Yahoo and is now a top-level Apache Software Foundation project. Its reputation is a lightweight but scalable way to provide event streaming.
  2. Appwrite — Appwrite is a secure end-to-end backend server for web, mobile, and Flutter developers packaged as a set of Docker containers for easy deployment.
  3. Terraform — Terraform is an open source infrastructure-as-code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies cloud APIs into declarative configuration files.
  4. Jamstack — Jamstack is designed to make the web faster, more secure and easier to scale. It builds on tools and workflows that developers love and brings maximum productivity.
  5. Argo — Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
  6. Knative — Knative is a Kubernetes-based platform used to deploy and manage modern serverless workloads. Google’s recent submission of Knative to the CNCF Sandbox should provide a lift to the project.
  7. TriggerMesh Cloud Integration Platform — It’s an open source event-driven integration platform powered by Kubernetes and Knative. TriggerMesh allows you to declaratively define event flows between sources and targets with content-based filtering, splitting and processing using functions.
  8. Kong — Kong is a very popular API gateway. Kong is built for hybrid and multicloud, optimized for microservices and distributed architectures.

These are just a few examples of high-quality open source applications that can improve productivity and effectiveness for cloud operators, DevOps and cloud native developers.

A Cloud-y 2022

The forecast for 2022 is cloudy. The companies that have weathered the pandemic best have leveraged cloud technology to increase productivity. Others did so to help modernize their business and survive a more socially distanced world. To that end, Gartner predicts global spending on cloud services to reach over $482 billion in 2022, up from $313 billion in 2020.

Also, since the publication of “The DevOps Handbook” five years ago, Google Trends has shown an all-time high for DevOps searches in 2021, and that is likely to increase through 2022. Combine that with massive open source growth illustrated in GitHub’s Octoverse report, including 16 million new users on GitHub raising the total number to 73 million and 61 million new repositories created. So we’ll likely have a very DevOpsy, open source and cloudy 2022.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Pragma, Docker.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.