Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.

4 Ways AI Will Shape CI/CD in 2021

how Artificial Intelligence and Machine Learning will help shape our software delivery processes and CI/CD pipelines in 2021.
Jan 11th, 2021 8:30am by
Featued image for: 4 Ways AI Will Shape CI/CD in 2021

Harness sponsored this post.

Tiffany Jachja
Tiffany Jachja is a Developer Advocate at Harness. Before joining Harness, Tiffany was a consultant with Red Hat's App Dev consulting practice. There she used her experience to help customers build their software applications living in the cloud. In her spare time, she likes to go on walks with her cat Rico and blog about self-development.

Continuous integration (CI) and continuous delivery (CD) are software practices that allow organizations and teams to deliver code to customers quickly, safely and repeatedly. Whether it’s to improve development, operations or security, CI/CD pipelines give engineers and teams more time to work on things that matter and less time struggling with the risk, standards or velocity of deployments. Software delivery, however, remains one of the most challenging problems enterprises face today. These challenges involve the cost of failure, outages, toil and misconfigurations.

The solution? Artificial intelligence (AI) and machine learning (ML) give engineers the data, automation and scale needed to deliver software value at an accelerated rate with confidence and sustainability. So in this article, we’ll share how AI and ML will help shape our software delivery processes and CI/CD pipelines in 2021.

AI-Driven Development

One of the greatest developments in 2020 for AI was the announcement of OpenAI’s GPT-3 model. These significant advances in natural language processing (NLP) open the door to new possibilities for how we develop code. The use cases can involve intelligent refactoring, automated unit test creation, and AI-augmented design.

In 2021, we’ll continue to see intelligent and low-code solutions emerge to accelerate development workflows and processes, inevitably changing the speed of software delivery.

Cloud Native Distributed Data Frameworks

Within the last decade, cloud native solutions have increased in popularity and adoption. This is no exception in the field of data science. Solutions like Ray and Dask offer cloud native support and flexibility for scaling and extending how data is processed.

In 2021, more workloads will move to the cloud and there will be a stronger ecosystem for cloud native application developers to integrate or use Big Data. Data Lakes and Data Warehouses, alongside synthetic data generation techniques, will also provide the utility for data-intensive applications — which often require high data quantities. The next series of CI/CD platforms and pipeline capabilities will leverage big data and analytics to provide deeper insights into delivery health and standards for software delivery.

AI-Driven Operationalization

Many solutions in the space today (like NewRelic, DataDog, HoneyComb and Splunk) provide additional insights into our software’s performance and quality. Monitoring and observability solutions allow organizations to better understand and define what is abnormal or normal in terms of application performance and function. In the past year, APM solutions, cloud providers and log aggregators have integrated new AI intelligence forms to analyze our software. These solutions provide information that will help shape continuous delivery, as CI/CD solutions integrate with different data sources to predict application or service defects, report that information, and take actions as part of the CI/CD process.

Next Level Insights into Delivery Health

DevOps and software delivery literature like the DevOps Handbook and Accelerate share practical approaches to continuous delivery; and understanding your team’s delivery health is the first step in improving a CI/CD process. AI and ML techniques can help correlate metrics to specific business goals and provide additional insights into the release quality of an application. Features like explainable AI allow systems to explain why a decision was made. New practices and regulations can improve the overall usage of AI/ML models within CI/CD pipelines. In a CI/CD pipeline, having visibility into our trained AI/ML models can help organizations continue to take appropriate data-driven actions.

The Future Is Ready for Intelligence

The most immediate need for AI in software delivery is for intelligent automation. So in 2021, we’ll continue to see organizations leverage and adopt AI and ML technologies and practices. We can all expect to continue the maturity of AI and ML capabilities within the technology industry.

This blog post shared four ways AI and ML will help shape and impact CI/CD in 2021. AI and ML enable organizations to move away from siloed operations management, provide intelligent insights, and drive automation and collaboration. If you’d like to learn more about Harness, the CI/CD platform, and how it uses AI and ML, click here.

Feature image via Pixabay.

Group Created with Sketch.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.