8 Best Practices for Container Secrets Management
Secrets management is a critical component of container security. In this post, we take a look at best practices for managing secrets (or, in other words, things like passwords, API keys and tokens) in order to mitigate container security risks and vulnerabilities.
1. Differentiate Between Secrets and Identifiers
A secret, in the context of containers, is any information that will put your organization, customer or application at risk if exposed to an unauthorized person or entity. As a system becomes more complex, the amount of secret information involved increases as well, and it follows that the level of risk involved increases. The more moving parts in the system, the more susceptible it is to risk.
It bears mention that not all information is secret information. Some information, like usernames and TLS certificates, are identifiers, and need to be shared selectively. Identifiers are not completely risk-free, but their risk is significantly less than that of secrets. However, passwords and keys associated with them should be treated as secrets.
2. Establish a Circle of Trust
There are certain parts of the system that can be entirely trusted (e.g., CPU, RAM, a root user or a secrets management tool with adequate security policies). Some parts of the system can only be trusted selectively (e.g., employees that are given explicit permission to access certain secrets or cloud services like an S3 bucket that is secured). Finally, there are outside entities that cannot be trusted by default. They include random users, or a WiFi hotspot. Secrets revealed to these entities are compromised and will lead to risk.
3. Gain Visibility into the Chain of Trust
As secrets pass through the system, they touch these various entities. The path a secret takes is key to deciding whether the secret is secure or compromised. Each step in the journey is a “link,” and together, the entire list of links forms a “chain.” It’s important to have full visibility across the entire chain. This way, you can establish a chain of trust, and ensure that only entities within the circle of trust have access to secrets.
4. Encrypt Data Using a KMS
A firewall isn’t adequate security for confidential data; neither is a password-protected database. They’re just one step away from being fully compromised. Instead, what’s required is a key management service (KMS) that encrypts data at multiple levels. You should be able to encrypt entire files with an encryption key, and individual pieces of data within that file using a different encryption key. This way, you can share only specific parts of the data without risking the rest of the data. This limits the blast radius of a potential attack. However, it is tedious to create and manage encryption keys on-premises. When working with cloud-native container apps, it is imperative to use a cloud-based encryption service like AWS KMS, or a similar alternative. They come with sophisticated features to automate and greatly control the creation and management of encryption keys.
5. Rotate Secrets Frequently
Secrets that stay the same for a long period of time are more likely to be compromised. As more users get access to a secret, it’s possible that someone has mishandled and leaked it to an unauthorized entity. Secrets can be leaked through logs and cache data. They could be shared for debugging purposes and not changed or revoked once the debugging is complete. They could be cracked by hackers. For all these reasons, secrets should be rotated frequently.
6. Automate Password Creation
The way passwords and access keys are created is vital to their security. Humans manually creating passwords is a recipe for disaster. According to Troy Hunt, 85% of passwords are insecure. The only way to combat bad passwords is to use automated machine-generated passwords that are unique and can’t be easily cracked. Today, most secrets management tools have automated password generation as a default feature.
7. Store Secrets Responsibly
Secrets management tools have become indispensable to container security. Their key focus is to prevent secrets from being saved on disk, embedded in code, or in any part of the system other than the secrets manager itself. It is a matter of culture, and is difficult to enforce across large teams. However, it is imperative to enforce the creation of passwords using the secrets tool, and the sharing of passwords using the same tool via tokens that expire.
8. Detect Unauthorized Access
Despite your best efforts, it’s likely that at some point a secret will be compromised. In this situation, all your prior planning will be tested. The first step is to identify the incident as early as possible (which is why you want a security monitoring tool in place).
You should also be prepared with a plan to respond quickly to a breach once it is detected. Let’s say you’ve identified a breach an hour after it took place. How fast can you move to disable access to the malicious user or entity? Here, full control over the passwords at every point of the system and a well-thought-out architecture are essential. Finally, you’ll need to change all passwords within the system as a precaution. Can this be implemented easily? Can legitimate users be informed once all passwords are changed, and can they be given access immediately? In case the system is offline, can the password manager still function?
Make sure you’re able to detect breaches, and that you have a plan in place for managing all of the considerations listed above.
Conclusion
Containers bring new security challenges for IT admins and DevSecOps teams. However, by being aware of the potential risks, and the available options to mitigate them, you can establish security that is more robust and almost foolproof.
