Diamanti sponsored this post.
Hybrid cloud is a reality — but it’s a complex one. Many enterprises have too much infrastructure to want to move entirely to the cloud, or they have applications they want to keep on their own systems for reasons of latency or data governance, while still wanting to take advantage of the flexibility and scale of cloud.
As Kubernetes increasingly looks like the de facto clustering standard for cloud native applications, organizations are making those same hybrid cloud decisions for Kubernetes workloads. What do they migrate to the cloud — or move from one cloud provider to another — and what makes sense on-premises? And crucially, how do they manage and monitor workloads across those different locations, migrating them to the best environment for cost, performance or security without needing to use multiple sets of tools to perform the same configuration multiple times?
Despite continuing advances, Kubernetes remains complicated infrastructure requiring expertise to deploy, configure securely and integrate with other enterprise systems. “There’s essentially no mass automation around how you think about managing large scale and large numbers of Kubernetes clusters, whether they’re on-prem or in the cloud”, says Diamanti’s CEO Tom Barton. A hybrid control plane needs to simplify that, on your own systems and in the cloud, and that’s the goal with Diamanti Spektra.
On-premises, Diamanti combines the simplicity of a full, turnkey Kubernetes stack with significant performance advantages from hardware acceleration — which fits the increasing trend to run Kubernetes on bare metal rather than in VMs for both cost and performance. It’s hyperconverged infrastructure, using Diamanti’s own Cloud Native Computing Foundation (CNCF)-certified Kubernetes distribution, plus their operational tools for logging and monitoring, and identity and RBAC integrations that work with LDAP, ADFS and enterprise systems using SAML or OpenID Connect.
Two PCI offload cards in each server handle storage and networking, so the CSI and CNI drivers can handle automatically provisioning logical volumes on physical drives when persistent storage is required, or allocating as many virtual network interfaces as required to given pods, for communication over different VLAN or VXLAN segments. Not only does that simplify deployment and deliver speed benefits (ten to 30 times faster than a system like Nutanix, according to Barton); the deep integration also enables intelligent scheduling based on available resources, with advanced storage and networking options like transparent SSL offload termination, synchronous mirroring, encrypted snapshots and data migration across nodes or availability zones.
“We can smooth over the rough edges and decision points required in a Kubernetes DIY solution today,” explains Brian Waldon, VP of product at Diamanti. “Then we provide additional value by mapping directly to business objectives; the ability to provide say 100,000 IOPs to a volume maps to a higher level SLO.”
More Clusters, More Clouds
Spektra extends deployment and application management from single to multiple cluster deployments — including Kubernetes clusters that aren’t on Diamanti infrastructure — whether they are on private cloud, on other bare metal servers on premises, or on public cloud Kubernetes services.
“Kubernetes really is the new Linux here — it’s providing an abstraction around literal operating systems — and we’re able to build reusable services and reusable tools that work on top of Kubernetes, independent of the infrastructure provider,” Barton said.
Spektra handles cluster creation and management, data services and workload provisioning, and offers the same storage and network features as the on-prem system — although without the hardware offload. It also handles the same operational services like logging and monitoring, on other Kubernetes platforms, without needing multiple tools — or having to go to multiple vendors with support questions.
Having a single control plane avoids both management overhead and what Waldon calls “mismanagement because of repetition.” “Being able to bring the dev, QA and production clusters under one pane of management, and then accurately hand out authorization rules based on the types of environments and the services they’re operating maps directly to business requirements. We need the SRE team able to access all three of these clusters, so being able to log and manage your identity and your backend authorization policies are incredibly important.”
Spektra aims to combine the flexibility and scale of cloud Kubernetes with policy-driven deployment. “Customers can deploy Kubernetes applications based on policies, based on access to features, based on different operational guarantees, or even something like data governance,” Waldon said. “Customers can model application requirements, and then automatically select the appropriate infrastructure.”
Continuous Cloud Motion
As well as deployment and management, Spektra handles workload migrations, which will become increasingly common in operational workflows.
“Because we’re sitting in the data path, we’re able to replicate data at the block level out to cloud providers, so we can provide disaster recovery, completely transparently behind the scenes, from an on-prem cluster into a cloud-hosted Kubernetes cluster,” Waldon points out.
“This is not about moving applications and workloads among clouds based on ‘penny by penny’ or ‘minute by minute’ changes in price. If a hurricane is forecast in one region, mobility enables an application to be moved out to another region, and back once it has passed. Indeed, disaster recovery is the primary use case for hybrid cloud — an app may move between development, staging and production across different cloud instances operated by different groups, or between partners. This is not repatriation or a ‘boomerang’ effect, it’s more like a revolving door, and we expect this kind of mobility to become a normal part of IT activity.”
You don’t need Diamanti infrastructure to use Spektra; it’s available as a hosted service that organizations can try out in preview now. General availability is planned for early 2020. “This would allow a customer to offload the responsibility for high availability or fault tolerance of their control plane,” said Barton. “If their on-premises infrastructure may not necessarily have a 100% uptime guarantee, they’re able to rely on us to operate that control plane for them.”
Once you have a hybrid management plane for Kubernetes, and as a hybrid cloud goes from being an ad hoc reality based on the choices of different business teams to a deliberate operational decision, organizations will be able to use the scale and commodity pricing of cloud for scenarios like machine learning training. Future GPU and Kubeflow support in Diamanti will cover both on-prem and cloud clusters (and other accelerators like Google TPUs), again on the same control plane, Waldon suggests.
“Because we implement the storage solution, because we replicate out to the cloud, we can directly facilitate cloud bursting for a use case like training in say GCP, and then bringing data back on-premises once your models are ready to deploy into production environments,” Waldon said. And those kinds of hybrid scenarios will require a common control plane.
CNCF is a sponsor of The New Stack.