How Kubernetes Policies Work (and When They Don’t Scale)
Kubernetes is a system for managing and orchestrating containerized applications in a cluster.
There are a lot of dynamically moving parts in such an architecture. To help you keep tabs on them, Kubernetes uses the concept of “policy.”
In this post, I’ll explain what Kubernetes policies are and how they can help you manage and secure your cluster. I also discuss their limitations and why they are difficult in some cases to use at scale.
Kubernetes Policy Basics
The purpose of policies in Kubernetes is simple: They define settings that should be applied across the cluster. That doesn’t mean that they are blanket policies; on the contrary, Kubernetes policies are configurable in many ways. You can tell them to make Kubernetes do different things to different types of users, for example.
But at a high level, Kubernetes policies serve to create and enforce cluster-wide settings.
Currently, Kubernetes policies can be used to govern several distinct realms. Networking, resource consumption and identity and access management are the major ones.
Kubernetes Network Policies
At a low level, Kubernetes has network policies, which is a construct that manages traffic at the pod layer and controls traffic coming in and out of pods. Namespaces can help in organizing resources, but they do not provide isolation.
By default, every pod can connect with other pods, and all traffic is allowed on every service. This is where network policies play a role. A network policy specifies how selections of pods can communicate with each other and with other network endpoints. It is defined by a pod selector (to indicate which pods it applies to), a list of ingress rules (to indicate which inbound traffic is allowed) and a list of egress rules (to indicate which outbound traffic is allowed). A NetworkPolicy resource uses labels to select pods and then define a list of rules that determine what type of traffic can reach the selected pods.
Kubernetes Resource Quotas
There are also policies for resource consumption. These are tools to address concerns around resources being consumed in a shared cluster. A resource quota can constrain resource consumption per namespace and limit the number of objects that can be created by type and how much computing power (i.e. CPU, memory) may be consumed in each project.
Limits can also be set on the number of storage resources requested in a given namespace. To implement this, create a ResourceQuota object per namespace, and the system will track usage in each namespace to ensure it abides by the rules defined in the ResourceQuota object. The support for this is enabled by default for many Kubernetes distributions — and note that these quotas are independent of cluster capacity (i.e. if you give out more quotas than you have resources, you can run out of capacity, even though the quota limit has not been reached).
Kubernetes Security Policies
Of high importance is understanding how security policies work in Kubernetes. This mostly involves authentication (verifying identity of a user) and authorization (specifying what they are allowed to do).
There are multiple ways to grant permissions in Kubernetes, called authorizers. These include node authorization, attribute-based access control, webhook and role-based access control (RBAC). RBAC is the standard authorization mode in Kubernetes and allows fine-grained permissions, which are expressed as rules (a combination of verbs, resources, and resource names). For example, a “scaler” role may have permission to “update” deployments in a specific namespace in order to change their number of replicas.
Then there are Pod Security Policies, which allow for fine-grained authorization of pod creation and updates. They are also typically used to prevent containers from running as root or with privileged mode. It is an admission controller, which means it has the ability to add something after the authorization-oriented pieces have occurred. The PodSecurityPolicy objects define a set of conditions that a pod must abide by in order to be accepted into the system.
There are different ways to secure different parts of your stack. You can set policies around what container images get put into the production environment and manage how to set policies on what is already running in production. There isn’t a single technology that is setting the standard around how to do this and a lot of things live in the registry and in the CI/CD pipeline, and so you may have to consider strategies outside of Kubernetes as well.
The Limitations of Kubernetes Policies
Kubernetes policies are not without some setbacks. We will discuss what complications there may be in terms of applying them at scale.
First off, orchestrating your containerized application on Kubernetes is not a trivial matter. There are many dynamically moving pieces, and so it is difficult to obtain a comprehensive security solution. It can be tricky to set things up so that your users are restricted to do what they are allowed to do.
For example, what a specific user can do will be defined by their identity, the groups they belong to, the actions they can perform on various Kubernetes resources (pods, deployments, services, etc), the network and pod security policies that apply to the objects they create, etc. Thus, it is non-trivial to enumerate a specific user’s permission and their reach, and the task has to be re-done each time one of these components is updated, and for each user.
Moreover, the current policy components in Kubernetes are all in different degrees of maturity. Some policies are extensible within each component, while others are not. The languages used to express intent by each project also vary. A consensus across the entire Kubernetes domain has not been established. It hopefully will be someday, but for now, this lack of maturity is another major limitation for Kubernetes policies.
Current Status
To address these issues, a Policy Working Group has been established with the intention of providing an overarching view of policy in Kubernetes. Kubernetes is divided into special interest groups for various topics. A working group goes across SIGs, and the one for policy will go across the SIG architecture group. The aim is to describe both the current policy implementations as well as future proposals.
For now, to achieve a policy strategy that scales, you can rely on namespacing. Instead of giving access to a single pod or service, you can reason in terms of namespaces. The namespace is a feature that allows a container to have its own IP address and routing table. When a container is created, there is a network namespace created for a container and it can be shared between containers.
Open Policy Agent is an interesting project as well, which enables Kubernetes administrators to have fine-grained policy-based control across their stack.
Conclusion
Kubernetes policy is a very powerful and useful feature that impacts security, but there still remain challenges. Scaling policies are not without problems because of the complex nature of deploying applications with so many moving parts. There are many things to secure, each in a different way, and it can be easy to overlook something. So we need checklists, and they tend to be lengthy. We also need to consider organizational culture and make sure to give the right privileges to the right people, and we also need a system to easily revoke such privileges.
For now, there are useful communities for addressing topics on policies and emerging projects, and private enterprise solutions to improve policy implementation at scale. Following certain security-minded companies on Twitter is also helpful.