Inside the Mind of a Kubernetes Attacker
Cloud Native Computing Foundation sponsored this post, in anticipation of the KubeCon + CloudNativeCon North America 2020 – Virtual, Nov. 17-20.
A framework for understanding attacker tactics and building your defenses.
New vulnerabilities in Kubernetes are discovered on a regular basis. In fact, 17 CVEs — Kubernetes and Kubernetes-related vulnerabilities — have already been discovered this year. Left unchecked, these vulnerabilities can become the means for attacking your cloud native applications — leading to catastrophic container escapes, privileges escalations or complete takeovers of entire clusters, and eventually, exfiltrations of sensitive data or parasitic cryptomining operations.
The good news is that these vulnerabilities can be patched. The bad news is that the Kubernetes threat landscape is a moving target and is constantly evolving. There will always be new, known or unknown, vulnerabilities that attackers will try to exploit. That’s why to stop Kubernetes attackers, patching your Kubernetes environment and staying up to date on the latest vulnerabilities is a great start — but ultimately insufficient. You need to think like an attacker and build your defenses to anticipate their attacks.
Fortunately, there are frameworks that can help you to get inside the mind of an attacker. In April, Microsoft Azure released a version of the MITRE ATT&CK framework adapted to the unique security concerns of Kubernetes. The threat matrix identifies the tactics used at various stages commonly found in cyberattacks. Below, you can see an expanded version of Azure’s Kubernetes threat matrix that includes several new tactics left out of the original Matrix — some of which were exposed by recent CVEs (the new tactics are in bold).
Each column in the matrix represents a different attack stage, with the items in each column representing tactics deployed by attackers in that stage. Conversely, each column also represents an opportunity to layer your defenses. For example, if an attacker gains initial access to your environment through a compromised user endpoint, you can still mitigate privilege escalation through reducing or eliminating role-based access control (RBAC) permissions. Layering your security in this way prevents any one weak point from compromising your cluster.
For a better sense of how to use the matrix, let’s use the CVE-2020-14386 vulnerability discovered last month as an example. The vulnerability exploits a memory corruption in the Linux kernel to gain root privileges from unprivileged processes. In a containerized environment that shares the host operating system kernel, this can lead to container escape — opening the door to multiple threats, ranging from exposing sensitive data, pivoting to other assets in the cluster, or threatening system availability.
In a container-based system, containers typically share the host kernel. So in this scenario, a compromised container exploits the Linux kernel to escape the container sandbox. In this specific vulnerability, an attacker crafts network traffic and sends it out through the container network interface. Once the traffic hits a certain kernel code, it can cause memory corruption in the kernel that could result in container escape. Interestingly enough, it should be noted that Google’s gVisor and hypervisor-based sandboxes are immune to this specific vulnerability.
But to exploit this weakness, the attacker would first need to gain initial access to a container that meets the prerequisites of this attack; which in this case is the ability to craft and send network traffic. An attacker could do this by compromising an image registry, for example, or by exploiting an application vulnerability. Other initial access tactics are found in the first column of the matrix.
Once initial access is gained, the attacker can persist the attack and move on to the next asset by planting a persistent Docker container directly on the host, that would run even if the system was rebooted — which could, for example, run crypto-mining software. They could also persist the attack by installing a mutating admission controller that injects malicious SideCar containers, or by rewriting container life cycle hooks to run scripts before the container is spun up. These persistence tactics can be found in the third column of the matrix.
In the case of CVE-2020-14286, if the container is running the CAP_NET_RAW capability (which is the insecure default configuration in Kubernetes, so the permission needs to be explicitly reduced), the attacker can exploit the hosting system and escalate privileges to escape the container. By exploiting another CVE, the attacker may be able to escalate privileges from kubelet (the Kubernetes node agent) to the entire cluster. From there, the attacker can steal Kubernetes secrets. Privilege escalation tactics can be found in the fourth column.
Using the Kubernetes Threat Matrix to Build Your Defenses
The native Kubernetes security controls are a good place to start for mitigating threats. First, you want to establish a solid security foundation to prevent initial access. This includes strong identity authentication mechanisms to manage access to your Kubernetes cluster, and strict RBAC privileges to perform operations against the cluster — being careful not to overprovision privileges to people or automation systems that don’t need them.
You also want to lock down the software supply chain. Images should be pulled from a single or a very specific set of container image registries, to minimize the software origins that can run inside the cluster. Also your image registry push policy should be screened with code scanning and container image scanning. This would prevent the initial access through the corrupted image registry previously mentioned in the context of CVE-2020-14286.
In runtime, you should minimize the capabilities and privileges of your containers, leaving attackers with fewer options to escalate attacks. Generally, application workloads shouldn’t be root or have access to the API server unless there is a specific need for it. In the example of CVE-2020-14286, this would involve dropping the ability of containers to use CAP_NET_RAW sockets — or any for that matter — which prevents the process from ever mounting the attack. This case illustrates how tuning your configuration as early as possible in development, via CI or CD, can dramatically reduce attack surfaces in your environment.
You also want to prevent the sprawl of sensitive assets, particularly with secrets. Last year we found that 89% of Kubernetes deployments are not leveraging secret resources properly. This leaves sensitive information — such as API keys, tokens and passwords — wired in the open, for any attacker with access to read Kubernetes API server resources. Properly managing and using secrets can go a long way towards hardening your cluster security.
To keep up with newly discovered vulnerabilities, make sure you have a playbook for patching and upgrading your Kubernetes infrastructure. If you are using a fully hosted environment, such as EKS, your cloud provider will patch your clusters for you. But if you are managing your environment yourself, you should have a plan to regularly upgrade or patch your clusters.
Unfortunately, there will always be undiscovered vulnerabilities that can wreak havoc before they are patched. Monitoring your runtime environment can go a long way towards mitigating these threats, and prevention can even go further than that. Analyzing your cluster API server runtime activity and application workloads network activity, as well as building adaptive models that profile user and application behavior, are critical components to complement the native security controls and policy-based security controls of Kubernetes. Effective monitoring can also overcome the defense evasion and persistence tactics found in the threat matrix.
When implementing security measures, don’t forget to be mindful of the people responsible. Your security team should be large enough and skilled enough to handle the responsibility delegated to them. For example, CVE scans can create a mountain of alerts that small teams may struggle to work through. It’s important that your runtime security controls, security tool stacks and incident response workflows match the skills of your team. Fortunately, automation can ease the burden of managing security controls on your team and make your life easier.
No matter what your team’s security skill set is, you should always start by putting yourself in the attackers’ shoes. What are your most important and sensitive assets? What hoops do attackers need to go through to get to them? Once you identify those two components, you can build your defenses in layers along their path through the Kubernetes threat matrix.
To learn more about Kubernetes and other cloud native technologies, consider coming to KubeCon + CloudNativeCon North America 2020, Nov. 17-20, virtually.
The Cloud Native Computing Foundation and Red Hat are sponsors of The New Stack.
Feature image via Pixabay.