The Secret to Securing Kubernetes
There’s no doubt that containers have officially entered the mainstream. Today, one in four enterprises allocate more than $250,000 a year to these technologies.
Just six years since inception, Kubernetes has become one of the most popular and powerful open source platforms for container orchestration. It automates many of the processes required to start and stop the various containers that power an application and automatically distributes containers across different host servers within a cluster and balances load between them. For organizations looking to orchestrate containers on premises, in the cloud or in hybrid environments, Kubernetes is the natural first choice. In fact, all of the major public cloud providers now support a conformant version of the platform.
As Kubernetes settles into maturity, conversations around cybersecurity risk continue to evolve — in large part due to aftershocks of CVE-2018-1002105, the most severe Kubernetes vulnerability to date. Meanwhile, the definition of “privilege” has expanded beyond powerful human users to encompass a massive array of machines and applications in these environments. Further, ordinary user secrets — like credentials, passwords, SSH Keys, certificates and API tokens — for Kubernetes can potentially hold as much power as administrator-level credentials for traditional systems. In the wrong hands, these secrets can be unlocked to spell catastrophe.
Kubernetes developers and security teams alike increasingly recognize the need for stronger security practices, but may need some help getting started. Here are five proven practices for securing Kubernetes — along with some pitfalls to avoid.
Secure Your Secrets
All of the containers and pods within Kubernetes need to interact with each other to do their jobs. For this to happen, they have to share secrets to securely access resources. Similarly, human developers are often provided secrets to gain high-level access to all containers and pods within the environment. If these secrets are mishandled or inadvertently exposed, attackers can gain root access, use bot crawlers to search out exposed secrets, then exfiltrate sensitive data, maliciously destroy data or crypto-jack cloud resources. For example, in the 2018 Tesla breach, cyber attackers discovered an unprotected Kubernetes console that contained AWS access keys. The attackers were able to copy the keys and use them to install cryptocurrency mining software in the organization’s cloud.
As DevOps teams build and deploy applications at scale, automation generates secrets. The result: even more secrets. The key to protecting your Kubernetes pods and containers is to continuously manage and secure both human and machine access to these powerful secrets.
Most organizations already have established requirements for securing credentials in traditional IT systems such as Windows. Similar practices should be implemented for securing secrets in Kubernetes: Authorize access based on a strong security policy (Role-based access control, or RBAC, works well in these elastic environments); rotate secrets regularly and consider implementing just-in-time temporary access to the highest-level secrets; and audit access to critical systems to minimize unauthorized access. For human users, consider keystroke or video logging to instill accountability and demonstrate compliance.
Mandate MFA
Cyber attackers are constantly evolving their methods. Recently, threat researchers have observed a swath of attacks targeting Kubernetes clusters by exploiting RBAC permissions. When successful, these attacks can compromise applications or abuse resources for malicious purposes.
These attacks remind us that while RBAC is a solid way to ensure proper access authorization, it isn’t a silver bullet. Some secrets providers protect secrets by requiring authentication via… another secret. That means that if an attacker gets their hands on that secret through RBAC, they can open the first door, get the key, open the next door, and so on, until they reach their destination.
To minimize risk, implement strong multifactor authentication (MFA) for any human or machine trying to request secrets. This way, a stolen secret will have no value to an attacker without the authentication of additional attributes such as random UUIDs, cryptographic keys or certificates generated with Kubernetes APIs.
Keep Your Secrets, Secret
To err is human. And it turns out applications don’t have a great track record either.
The best way to keep your secrets secret is to isolate them from applications and developers while still giving them the access they need. By using a system that acts as an intermediary, human developers and applications alike never have to see the actual secrets, and therefore, can’t expose them.
Applications can securely connect to databases, services and other protected resources without fetching or managing secrets. Meanwhile, this “secretless” approach makes it easier and safer for developers to write code for their applications: They don’t have to write API calls to fetch the access secret from the Kubernetes secrets store, and they can access resources without resorting to insecure practices like hardcoding credentials. Open source tools that work within Kubernetes make it easy to take this crucial security step.
Standardize for Simplicity
It’s true Kubernetes has its own built-in capability for handling secrets, aptly named Kubernetes Secrets. Using this capability, sensitive data such as passwords and keys can be stored in a Kubernetes Secret Object. This information is stored in etcd, which is not encrypted by default, but etcd can be encrypted at rest with basic access policies. This functionality offers basic security of secrets, as is the case for many of the tools and services offered by cloud vendors and container platforms. The problem is that each native tool approaches security differently, has a varying level of maturity and uses a different interface.
If you’re like most digital businesses today, you rely on multiple DevOps tools and platforms — from Kubernetes and Chef, to Ansible and OpenShift, to Amazon Web Services and Docker. So, with a different secrets management capability for every tool in your constantly evolving toolset, what could possibly go wrong? A lot, actually. Integration can be painful (or downright impossible) and each tool establishes its own independent “island of security,” which can lead to dangerous gaps.
Establish a centralized, standardized secrets management system to control, enforce, monitor and manage access across your applications, clouds and tools. This will help your organization maximize operational efficiency, strengthen security, meet audit requirements and reduce the risk of damaging security breaches.
Find Your Blind Spots
You can’t secure what you can’t see — or what you don’t even realize exists.
Discovery tools can help you identify unsecured secrets across your entire cloud and DevOps environment and uncover previously unknown vulnerabilities. You can also take advantage of open source tools that can pinpoint roles, users and pods cyber attackers could exploit to compromise entire Kubernetes clusters and create far-reaching damage. These can be particularly helpful for identifying potential weaknesses across large environments, giving administrators the visibility they need to reduce risk.
As with all great open source projects, Kubernetes will continue to evolve, shaped by a diverse blend of voices, visions and expertise. It’s encouraging to see security rise to the forefront of the conversation as the platform progresses. By taking these five steps, your organization can bolster overall cybersecurity, drive developer efficiency and speed, and ultimately, help the Kubernetes community at large withstand the test of time.
To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam later on this year.
