Reducing Security Vulnerabilities in Kubernetes
In this post, we discuss critical container security best practices to help you avoid complicated security challenges and successfully reduce vulnerabilities in your containerized environments in Kubernetes.
Containerized environments offer many advantages to developers but also pose complicated security challenges. Following container security best practices can help you mitigate risk, reduce vulnerabilities and successfully deliver verified software.
Here, I’ll provide an overview of container security fundamentals for developers. If you’re a developer creating a Docker container and want to ensure its security, the following reminders can help.
No, Namespaces Aren’t Enough Isolation
A namespace is a virtual cluster within the main Kubernetes cluster that provides a mechanism for isolating groups of related components within the control plane. However, assuming namespaces are enough for resources to execute processes in an isolated environment is where most developers are mistaken.
While namespaces — and the resources running in them — give you additional context and isolation within them, they don’t isolate your resources from other namespaces.
So, you’ll need a namespace as a beginning to describe the deployment or service, followed by additional configuration for the isolation to be effective.
Also, pods can continue communicating with each other via their IP address, regardless of the namespace they’re in. This is because cross-namespace communication is a default setting in a Kubernetes cluster and the usual setup.
Don’t Container and Root
Following the principle of least privilege (PoLP), the main difference between root and non-root containers is that the latter are focused on ensuring the minimum privileges necessary to run a process. If developers stuck to this principle as much as possible and didn’t run containers as root, there would be fewer serious security issues.
Let me explain: You have a service that’s listening on a privileged port, say port 80, that you need to run as root. But you should also ask yourself whether you really need to run your service on port 80. Your responsibility is to make this decision after being aware of all the possible implications.
“But it’s running in a container — shouldn’t that be fine?”
Unfortunately, no. This is because of a critical security problem: privilege escalation.
Preventing privilege escalation is a key argument to avoid running a container as root. A root user inside a container can run every command as a root user on a traditional host system, including starting services, installing software packages and creating new users. This is, of course, undesirable from an application perspective, and it applies to containers as well.
You also need to be aware of container runtimes or kernel vulnerabilities. The root user can access and execute anything on the underlying host as a highly privileged user. This means they’ll have access to usernames/passwords that are configured on the host to connect to other services, can install unwanted malware and access other cloud resources, among other activities that may put file system mounts at risk.
Most services can run without escalated privileges. For those that can’t, there are capabilities that give you fine-grained controls. This is specifically true for privileged containers, which represent the highest escalation level with all the available capabilities. If somebody were to take over a service in your container, that person would need to have the highest privilege available. Otherwise, they simply couldn’t. So even if a containerized app really needs some advanced permissions, you still shouldn’t run it as --privileged; it’s better to use --cap-add for more fine-grained control.
Docker also has a rootless mode that you can use to run Docker daemons and containers as a non-root user. What’s more, you can use this mode to mitigate vulnerabilities in daemons and container times that sometimes give malicious agents root access to entire nodes and clusters.
Another important note here is that a rootless mode is capable of running daemons and containers without root privileges by default within a user namespace.
Networks Need Segmentation
Using an intrusion prevention system (IPS) to block malicious traffic isn’t enough to adequately protect containerized environments — and I say this from experience.
Let me illustrate this with a brief anecdote. I used to work at a company that implemented the flat network model for the sake of collaboration. As you may know, flat networks and accessible areas mean everyone in the network has access to everything, which can present significant security issues. I asked the network designer at the time if they were concerned about this, and they told me they had “implemented an IPS to prevent malicious attempts.”
The architect left the company a year later, and a big security breach followed their departure. Simply relying on an IPS wasn’t good enough then, and it certainly isn’t good enough now.
So what can you do to avoid breaches? Build network segments. The main advantages of this are the following:
- They minimize breaches — By creating network segments containing only the resources specific to the consumers that you need to authorize access to, you create an optimal environment of least privilege that significantly minimizes breaches.
- Minimize damage — They contain the blast radius in case of successful exploitation. This makes it easier to prevent lateral movement from a container that has been exploited by malicious agents and to comply with PCI DSS standards.
- Minimize data exfiltration — Network segmentation also helps prevent or minimize the bad effects of data exfiltration, an area where an intrusion detection system (IDS) and an IPS fall short. As they are reactive, IDS and IPS can only detect a new attack or vulnerability when rules have been updated on these devices. As such, there is always a chance the update could come too late, making it easier for cybercriminals to find and exploit your networks.
Keep Your Dependencies up to Date
Dependency management is a crucial aspect of creating and maintaining a secure and reliable software supply chain. It’s also slightly complicated because of the presence of daemons and other factors that make it difficult to determine what exactly needs to be done to keep dependencies up to date.
That’s also why simply using the latest tag for Docker containers isn’t enough to ensure adequate protection. To do this right, you have to first know your dependencies by:
- Looking at the dependency checks in your CI/CD (npm or Apache Maven).
- Monitoring relevant data sources such as security advisories.
- Updating/rebuilding regularly (this helps to ensure the latest security patches are in place).
You also need to be careful with “hidden” or embedded dependencies. When you run a Docker container, you’re likely to use a package manager to install dependencies. Sometimes you might have to download something like modules or even build a piece of software yourself. In both cases, you should be sure to update these dependencies because they don’t necessarily pop up in all scans and can become a potential container security concern if they become obsolete.
Another important aspect of dependency management is to only use trusted sources and be certain the project at hand is an actual Docker Hop project.
For starters, you need to be aware of typosquatting, where scammers use misspelled or modified domain names to trick users into visiting fraudulent websites. It’s shocking how many projects out there will try to give you malicious software if you use, say, “MGINX” instead of “NGINX.” So be careful with what you type and click.
Other best practices include using descriptive tags, instead of the latest tags, and reviewing dependency updates regularly.
Takeaways
Container security involves continuously implementing and maintaining security controls to protect containers and the underlying infrastructure.
By applying the above best practices to your development pipeline, you can improve visibility into containerized workloads. This’ll help you secure all container resources and components from the initial development phase until the end of their life cycle, preventing breaches and ensuring timely remediation.
Here are a few resources to help you brush up on your container knowledge: