Kubernetes Security Best Practices to Keep You out of the News
Kubernetes can look enticingly powerful at first glance, like it will solve all of your problems. (It solves many of them!) But anyone working with it knows how quickly things can get complex. Kubernetes security is no different.
Kubernetes is not secure by default. There are multiple attack pathways and CVEs are regularly found. While it can feel overwhelming to get started, there are some concrete tactics you can implement to protect your services and infrastructure — and keep your organization out of the headlines. Learn the major components of the pipeline you’ll be securing, then how to secure them.
First let’s get an overview of the ecosystem you’re securing.
- Containers. Kubernetes is a container orchestration system. So any attempt to secure Kubernetes must involve securing the containers it deploys, including the pipeline of building and deploying them.
- Linux and Windows. Containers usually contain an operating system (OS), in addition to the OS powering the servers or VMs that Kubernetes is deployed on, so we have to talk about securing the OS, whether it’s Linux, Windows — or both.
- Kubernetes. Kubernetes itself is both an API server, as well as a distributed system of agents and etcd databases within a network — all of which also need to be secured.
Going with a managed Kubernetes solution such as those offered by every major (and minor) cloud provider can often get you better default configurations, or easier to manage features; but none of them are inherently secure without choosing the right options.
For example, the most secure way to authenticate to any Kubernetes cluster is using OAuth. But generally, authentication via a private client certificate file and/or a username and password are both turned on by default. Conversely, configuring a Container Network Interface (the internal service that facilitates container and pod communication across the Kubernetes network) that supports Network Policies (discussed below) can be a hassle, but most managed offerings make it a click-button option.
Let’s see how to secure each of those components in turn.
Containers
Containers need to be built securely, and the container images deployed must be trusted to run in the system.
Building
Building secure containers requires scanning them for vulnerabilities — including Linux system packages, as well as application packages for dynamic languages like Python or Ruby. App developers might be accustomed to scanning application dependencies, but now that they are shipping an entire operating system with their app, they have to be supported in securing the OS as well.
To support this effort at scale, consider using a tool like Cloud Native Buildpacks, which allows a platform or ops team to make standardized container builds that developers can use to drop their application into — completely replacing the Dockerfile for a project. These centralized builds can be kept up-to-date so that developers can focus on what they’re good at rather than having to be jacks-of-all-DevOps-trades.
Container image scanning tools scan the layers of a built image for known vulnerabilities, and are indispensable in keeping your builds and dependencies up-to-date. They can be run during development and in CI pipelines to shift security practices left, giving developers the earliest notice of a vulnerability. The best practice is to strip your container down to the minimum needed to run the application. A great way to ruin an attacker’s day is to have a container with no shell!
Signing
So now you’ve built secure containers. But how can you be sure that the containers you built are the ones that are actually pulled into your cluster? Docker supports signing images with a key, which can then be authenticated at the time of pull and deployment. Signing containers is similar to adding TLS certificates to endpoints; it prevents man-in-the-middle attacks by allowing you to verify that the containers images you are pulling are the exact ones you pushed. For this to work, you need to have systems on both the pushing and pulling sides aware of the same key. We’ll see how to prevent unsigned images from deploying into your cluster below, when we examine Admission Controllers.
Linux
Your containers are probably running Linux or Windows, and Kubernetes is probably running the containers on one of those as well, since it supports a mix of Linux and Windows worker nodes. To ensure that your system is secure, you must still do the traditional work of making sure servers are only exposed publicly when they have to be, that SSH credentials are secure, that OS libraries are up-to-date, and that user and group permissions are locked down. If an attacker can get access to your master or worker nodes, it’s much easier for them to compromise any part of the Kubernetes system — be it the API or the kubelet agents. Even in a cloud native world, there’s still a need for good old sysadmin work, whether you delegate it to your cloud provider or do it yourself.
Kubernetes
Fully securing Kubernetes can, and has, taken up a book, but for this discussion, the most critical aspects are role-based access control (RBAC), admission controllers and network policies. Additionally, cloud native security platforms can help fill in the little gaps that attackers love to squeeze through.
RBAC
Who can do what in your cluster? Role-Based Access Control (RBAC) answers that question. Kubernetes provides the ability to give specific permissions to users and service accounts, in the cluster as a whole and in a given namespace. Some example use cases are allowing all teams to view each other’s application specs but only be able to modify those in their dedicated namespaces, or allowing your CI/CD pipeline agent to deploy things to production and only a few vetted individuals to delete things there. These will vary by organization, but actively managing RBAC is essential to a secure cluster.
Admission Controllers
Admission controllers (AC) are the only way to achieve full Kubernetes security. ACs prevent containers from being scheduled and run in the cluster unless their pod specification fulfills certain criteria. There are lots of kinds of ACs, but two stand out for review: the PodSecurityPolicy AC, and any AC that validates signed images.
PodSecurityPolicies (PSPs) stand between attackers and the most vulnerable aspects of the Kubernetes system. For example, containers in Kubernetes can ask to mount any path on the underlying host using a “hostPath” type volume, such as the Docker socket. A container that mounts the docker socket can, without privileged status, run any docker command on the host as root. That is scary! The only way to prevent this is a PSP disallowing hostPath volumes. There are several other profound attack paths that PSP settings can prevent. If you do only one thing to secure your cluster, it should be to create PSPs and enable the PSP Admission Controller.
Earlier, we discussed how to sign your container images to establish a trust chain. How does the chain complete? Using the Open Policy Agent (OPA) AC, you can have every container image checked for a valid signature before it is allowed into your cluster. This is a great guide on using Notary and OPA together to establish a full trust chain from build to deployment.
Network Policies
Kubernetes network policies are like internal firewall rules for the cluster, which should speak to their importance. They allow an admin to configure the cluster for scenarios like:
- One namespace can only talk to another namespace where its dependencies are.
- External traffic can only reach an API gateway container, and no other containers.
- Preventing egress from all containers except to a DNS registry.
The main limitation with network policies is that the container network interface (CNI) used by the cluster must support them. If you are managing your own cluster, not using a cloud-provider managed solution, you’ll need to research available CNIs to see what works for your network that offers NetworkPolicy support.
Cloud Native Security Platforms
The security landscape is always changing, and it changes even faster in the cloud native space, since it is evolving at an incredible pace. Even with all the above steps in place, it’s important to have something looking at your live runtime for anomalies and signs of a breach. Platforms like Prisma Cloud can alert you to such anomalies, and can actively prevent unexpected processes from running and network connections from being made.
This article has listed a number of open source projects solving each problem in turn, but when it comes to Kubernetes security, the “Build vs Buy” trade-off can quickly shift towards “Buy.” For small organizations, maintaining all of those tools can be burdensome; and for large organizations, running them at scale can bog teams down and burn valuable time and resources. A fully-featured platform can often save money in staff time and reduce the need for intervention by a dedicated security team, by making it easier to push security left.
Resources
For a longer look at the attack surface of Kubernetes, this Black Hat presentation by Ian Coldwater is excellent. The Thank Goodness It’s Kubernetes (TGIK) podcast on YouTube does 1-2 hour deep dives into various Kubernetes topics, including multiple episodes related to Kubernetes security.
Kubernetes security is a journey more than a destination. Hopefully, this guide has given you the tools to get started on it.
