Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
Containers / Kubernetes

A Self-Driving Cloud Platform for OpenShift

Red Hat’s OpenShift Container Platform is a great solution for deploying Linux containers and Kubernetes, but many organizations with diverse servers, storage and networking products face challenges implementing OpenShift.
Jan 3rd, 2019 8:28am by
Featued image for: A Self-Driving Cloud Platform for OpenShift

Red Hat’s OpenShift Container Platform is a great solution for deploying Linux containers and Kubernetes, but many organizations with diverse servers, storage and networking products face challenges implementing OpenShift. IT teams face challenges every step of the way, from deployment and setup to configuration, maintenance and upgrades, and those challenges increase organizational costs and sap resources from other projects. In addition, some enterprises can’t afford to hire infrastructure and operations specialists to attack those challenges. A self-driving private cloud platform can address these issues by providing an automated, centralized, web-based portal that centrally manages diverse infrastructure across multiple OpenShift sites.

OpenShift Challenges

Ajay Gulati
Ajay Gulati shares responsibility for driving ZeroStack’s product innovation, bringing to life his thought leadership in systems infrastructures and virtualization. Gulati was a senior architect and R&D lead at VMware where he designed flagship products including Storage I/O control, Storage DRS and DRS. He has been recognized as a prolific inventor by his peers and has more than a dozen patents to his name. Gulati has consulted with multiple startups including Nimble Storage and Velocloud Networks. Gulati has a Ph.D. from Rice University and a B.S. from IIT Kharagpur.

Like other software platforms, OpenShift requires server, storage and networking resources. Typical IT shops have separate teams for maintaining these infrastructure silos, and if the organization has multi-site operations, the process is made much more complex and labor-intensive. IT teams must perform a number of manual or otherwise often-unmaintainable scripted tasks in order to:

  • Provision the underlying infrastructure, both virtual and physical
  • Deploy the various components of OpenShift
  • Add/remove physical hosts for maintenance
  • Add/remove physical disks to increase storage capacity and/or failure management
  • Monitor resource allocation, utilization, and performance
  • Perform capacity planning

These are all time-consuming, error-prone, ongoing operational activities that are hard to automate at scale and harder still to manage in multiteam, multisite situations and in the face of failure conditions of various physical resource failures. OpenShift is not concerned with nor does it address any of the above challenges. Here are some other issues.

One OpenShift cluster doesn’t address all needs — Organizations have diverse applications teams, application portfolios, and sometimes conflicting user requirements. One OpenShift cluster is not going to meet all of those needs. Companies will need to deploy multiple, independent OpenShift clusters with possibly different underlying CPU, memory, and storage footprints. If deploying one cluster on diverse hardware is hard enough, doing so with multiple clusters is going to be a nightmare!

Development teams are often distributed in multiple sites and geographies — Companies do not build a single huge OpenShift cluster for all of their development teams spread around the world. Building such a cluster in one location has DR implications, not to mention latency and country-specific data regulation challenges. Typically, companies want to build out separate local clusters based on location, type of application, data locality requirements, and the need for separate development, test, and production environments. Having a central pane of glass for management becomes crucial for operational efficiency in this situation, simplifying deployment and upgrades in these clusters. Having strict isolation and role-based access control (RBAC) is often a security requirement.

Container orchestration is just one part of running cloud-native applications and infrastructure operations — Developing, deploying, and operating large-scale enterprise cloud-native applications requires more than just container orchestration. For example, IT operations teams still need to set up firewalls, load balancers, DNS services, and possibly databases, to name a few. OpenShift does not help with any of this.

Enterprises have policy-driven security and customization requirements — Enterprises have policies around using their specifically hardened and approved gold images of operating systems. The operating systems often need to have security configurations, databases, and other management tools installed before they can be used.

Enterprises need a DR strategy for container applications — Any critical application and the data associated with it needs to be protected from natural disasters regardless of whether or not these apps are based on containers. Existing solutions don’t provide an out-of-the-box disaster recovery feature for critical OpenShift applications; customers are left to cobble together their own DR strategy.

Private Cloud Architecture

A self-driving private cloud platform converts a cluster of servers into a web-managed cloud system that provides self-service consumption, monitoring, and an integrated learning engine for performance and efficiency insights. It consists of server virtualization, software-defined storage pools, software-defined networking with microsegmentation, built-in monitoring, and external storage integration. This solution enables multitenant, multicloud, and containerized environments.

Using Machine Learning and AI to Simplify Infrastructure Management

Traditional infrastructure management consists of ongoing checks on server, networking and storage resources. Often, the infrastructure is not fully optimized to support container-based operations, and multisite operations magnify the problem. A self-driving private cloud platform virtualizes these resources and can apply AI and machine learning techniques to automate capacity sizing.

A private cloud platform with AI-driven intelligence optimizes sizing, performs predictive capacity planning, and implements seamless failure management. IT operations can easily deploy load balancers, set up domain name system (DNS), implement virtual private network (VPN) services, and configure virtual firewalls within the private cloud. Since the cloud runs both Virtual Machines (VMs) and containers on the same platform, persistent data stores, SQL, and Postgres databases can be deployed into VMs and made accessible to containerized applications.

As part of its multisite capabilities, a private cloud provides remote data replication using external storage solutions between geographically separated sites, protecting persistent data and databases. In addition, users can connect multiple sites and private networks within those cloud environments using built-in VPN services and provide secure communication to enable fail-over scenarios at the underlying VM layer.

A multisite private cloud helps organizations with diverse application teams, application portfolios, and conflicting user requirements by offering logical business units that can be assigned to different application teams. Each application team gets full self-service capability within quota limits imposed by IT operations. They can automatically deploy their own cluster with a few clicks, independently of other teams.

Streamlining Container Deployment and Operations

Some private clouds include app stores that simplify operations with application blueprints that can be deployed within a few minutes. Users can extend the AppStore to licensed products like Red Hat OpenShift Platform Container by leveraging their own license or licenses provided by other partners. To simplify multisite management, users can reuse these templates to deploy the same configuration in different sites and locations.

For cost-effectiveness, an OpenShift platform should run on an automated, self-driving private cloud platform that abstracts hardware maintenance operations and costs. In multisite deployments, the savings in operations and infrastructure management costs can be as high as 90 percent over traditional infrastructure.

Red Hat OpenShift is a sponsor of The New Stack.

Feature image via Pixabay.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.