Virtual Clusters are the Future of Kubernetes Multitenancy
Sharing Kubernetes clusters across teams is extremely difficult. If you talk with people in the Kubernetes community, you’ll often hear them discuss how hard it is to get multitenancy right. This difficulty has led to many companies abandoning multitenancy entirely and instead provisioning dedicated clusters for teams or even individual engineers. However, more clusters mean more cluster management headaches for platform teams, and more energy consumption and negative consequences for the environment, as IBM‘s Holly Cummins mentioned in her keynote from KubeCon+CloudNative Con EU earlier this year.
Virtual clusters are a new approach that allows teams to share clusters without some of the pain involved with Kubernetes multitenancy.
State of the Art: Multitenancy with Namespaces
The primary approach to Kubernetes multitenancy is based on the construct of namespaces and strategies to isolate these namespaces. Teams or developers have a dedicated namespace in the cluster and are limited to that namespace by role-based access control (RBAC) permissions. Additional tools like admission control and network policies further limit what developers can do using the cluster.
One of the main stumbling blocks with this namespace sandbox approach is that some Kubernetes objects that developers would like to manage operate at the cluster level, not the namespace level. Say that a developer wants to deploy a Helm chart in a cluster that contains resources like services and deployments, which are namespaced, but the Helm chart also contains custom resources (CRDs), which are cluster-scoped.
That’s impossible in this namespace sandbox model because the developer will not have sufficient RBAC permissions outside of their dedicated namespaces.
The Idea Of Virtual Clusters
With a virtual cluster, a user is presented with a virtual representation of a cluster with its own control plane. It appears to be working exactly like a full-blown Kubernetes cluster, and its API requests are isolated from other users. A virtual cluster can run entirely within a namespace of a host cluster, depending on the implementation, and usually has a separate data store (think etcd) for objects that are created inside the virtual cluster.
In 2019, Darren Shepherd, Rancher’s chief technology officer, published an initial proof-of-concept implementation of virtual clusters called k3v, which started the journey toward making virtual clusters reliably work in Kubernetes. As of today, two working flavors of virtual clusters have emerged:
- vcluster, an opensource project maintained by Loft Labs
- Cluster API Provide Nested, a project driven by the WG Multitenancy community
These two projects have different takes on how to implement virtual clusters, but share the goal of enabling platform teams to reduce cluster sprawl.
Virtual Clusters to Address Multitenancy Challenges
There are several challenges platform teams operating multitenant clusters need to handle:
- How to isolate tenants?
- How to create a great user/developer experience?
- How to clearly divide responsibilities between platform users and platform operators?
Challenge 1: Tenant Isolation
As we mentioned, the standard approach for multitenancy so far has been namespace-based isolation of tenants using tools like RBAC, admission control and network policies. Security becomes a big concern when sharing resources like a Kubernetes cluster and the default security behavior is “deny all.” Platform teams operating a shared cluster will likely lock down user access to a namespace, and network policies can restrict which resources a pod can connect to, for example.
This deny-all approach can have a very negative impact on developers’ ability to get their jobs done, though. As they add functionality to their applications, they have to work closely with platform and security teams to get the right level of access for their pods.
How virtual clusters can help:
- Since virtual clusters generally just need to have the permission to create very basic objects, such as pods, in the underlying cluster, cluster admins can strictly lock down the cluster access for tenants to very few basic resources if these tenants work inside a virtual cluster.
- Separate control planes per user allow developers to send API requests that don’t affect any other teams working in the same cluster, providing an additional layer of isolation.
- Tools like admission control and network policy can be applied on the host cluster level, giving users more freedom within their virtual clusters.
Challenge 2: Developer Experience
If you’ve ever sat around waiting for another team to provision some infrastructure for you or to open a port on the firewall, you know how painful it can be. These delays not only hurt velocity, but they also reduce developer happiness. As Microsoft researcher Nicole Forsgren and her team mentioned in their excellent paper “The SPACE of Developer Productivity,” “productivity and satisfaction are correlated, and… a decline in satisfaction and engagement could signal upcoming burnout and reduced productivity.” Tooling matters beyond just cycle times.
Platform engineers know that the best approach for productivity is to enable self-service for users. Developers using Kubernetes clusters need infrastructure on demand, and they need the flexibility to deploy more than just pods. They may even need to work across multiple namespaces, for example, when testing several microservices together.
Also, developers in a company aren’t a monolith, and they may have different tooling needs. Some companies like Netflix recognize this, and while there are internal tools available for things like CI and deployment, teams can choose to use different tools that fit their use case. It’s tough to do that in a shared cluster, where tools running in the cluster can affect other teams.
How virtual clusters can help:
- Virtual clusters can be provisioned quickly and often autonomously by the engineering teams when they need them.
- Developers can deploy cluster-wide resources like CRDs into their own virtual clusters.
- Engineers can work across multiple namespaces in their virtual cluster without affecting other users.
- Developers can use whatever tooling they prefer to deploy to the cluster, from running
kubectl applymanually to installing a fully featured GitOps framework such as ArgoCD.
Challenge 3: Responsibility
Sharing responsibility for Kubernetes clusters can be contentious. Product engineers and platform teams can have very different goals and may be measured on opposing metrics. This misalignment is the reason DevOps emerged, but we’ve not completely solved this issue yet.
Platform teams generally understand that it’s to their benefit to expand developers’ permissions and empower them to take responsibility. They want to shift left to allow development teams to move faster, while maintaining and focusing on security and stability. But in this model of shifting responsibilities, many questions are to be answered by these teams: Who has access and authorization to perform a task and the expertise to do it? How long does it take for developers to get what they need? What can be automated, and who can automate it?
How virtual clusters can help:
- Virtual clusters clarify the responsibility model. Platform teams can manage the host cluster and are responsible for its stability. Developers own their virtual clusters running on top of the host cluster, and developers are responsible for what runs inside their virtual clusters.
- Virtual clusters are quick to spin up and tear down, so they allow teams to move quickly as new organizational structures emerge and responsibilities shift over time.
Conclusion
The momentum for virtual clusters is growing, and it’s easy to see why. Since we’ve launched our open source project vcluster, we’ve seen a lot of interest in the Kubernetes community for virtual clusters. The fact that the WG Multitenancy community has also been working on a virtual cluster implementation shows there is a compelling need for this kind of solution.
Operating multitenant clusters has been onerous for platform teams and frustrating for the people working in restricted namespaces inside these clusters. The promise of Kubernetes is immense, but the lack of support for multitenancy is a huge pain point. Virtual clusters solve real problems that platform teams and developers encounter every day. This is why we believe that virtual clusters are the future for Kubernetes multitenancy.
