Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
Cloud Services / Containers / Kubernetes / Microservices / Serverless

New Microsoft, Oracle Projects Bridge Kubernetes with Serverless Environments

Dec 15th, 2017 9:20am by
Featued image for: New Microsoft, Oracle Projects Bridge Kubernetes with Serverless Environments

At Kubecon 2017, held last week in Austin, both Microsoft and Oracle released new software to further integrate serverless capabilities into the Kubernetes open source container orchestration software, as well as to manage container operations that would not require user-managed servers to run.

Oracle has released as open source an installer that allows its serverless software, Fn, to be installed on Kubernetes; Also at the show, Microsoft introduced the Virtual Kubelet, which masquerades as a Kubernetes kubelet node agent to connect Kubernetes to API services outside K8s’ container environment.

In October, Oracle released the source code of the Fn Project, which was developed by Chad Arimura and other members of who joined Oracle after that company was purchased by Xenon Ventures. Forked from IronFunctions, Fn provides a way to execute functions written in many languages, by placing the code within a container. The software features a built-in load balancer and full compatibility with AWS Lambda function-as-a-service. Associated software Fn Flow, provides a way to build higher-level workflows and orchestrations, using the developer’s favored language and development environment.

The pairing with Kubernetes is useful in a number of ways, Arimura said. In a microservices environment, Kube-DNS can be used to do self-discovery of microservices from your functions, and vice-versa. Kubernetes provides a unified base for managing infrastructure, including serverless deployments.

The installer allows users in a Kubernetes network, such as the Oracle Container Engine Cloud Service, to tap into Fn serverless capabilities. The Fn installer has also been integrated with the Oracle Container Native Application Development Platform.

Microsoft’s work also draws from earlier efforts. In July, the company released its own Kubernetes connector for its Azure Container Instances (ACI) a serverless container runtime. The company expanded the scope of this connector, now called the Virtual Kubelet, to support other runtimes through a pluggable architecture of Kubernetes primitives.

“We thought of Azure Container Instances as a low-level piece of orchestration, and we want to use Kubernetes as the higher-level orchestration,” said Brendan Burns, who is a Microsoft distinguished engineer and one of the creators of Kubernetes.

Virtual Kubelet is a Kubernetes Kubelet, a small agent that runs on each Kubernetes node. This Kubelet provides a gateway to other API-led services. “Kubernetes API on top, programmable back,” is how the software’s GitHub page describes it.

The Virtual Kubelet has a pluggable provider interface to define actions that will be executed by non-container environments, including ‘serverless’ services. This opens the possibility of using on-demand and nearly instantaneous container compute services through Kubernetes API, without needing to manage — or be billed — the virtual machines or other supporting these services.

Kubernetes has the ability to schedule jobs, and so can call resources into being, allowing users to be billed only for the cloud time that is actually used to complete the work.  Intermittent workloads such as continuous integration and deployment development pipelines, or batch processing jobs would be ideal for this setup: “Anything where you don’t have something running all the time,” said Gabe Monroy, Microsoft Azure program manager for container operations.

The Kubelet is already integrated into the Azure command line interface (CLI), allowing Azure users to very quickly install the Kubelet into an Azure Container Service (AKS) cluster. Service provider Hyper is contributing the effort. That company had forked the connector to run secure containers, including the recently introduced Kata Containers, on its own cloud. At KubeCon, Burns and Monroy both spoke with other vendors who were also interested in harnessing the Kubelet for their own software as well.

“There are still a lot of interesting open questions left to sort out with Kubernetes and the scheduler, but I think we’ll have those conversations, not as ‘Azure’ but as a group that is interested in serverless containers,” Monroy said.


The Cloud Native Computing Foundation, which manages Kubernetes, and Microsoft are sponsors of The New Stack.

Feature image: Oracle’s Bob Quillin(L) and Chad Arimura at KubeCon 2017.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.