How Does HashiCorp Vault Work with Kubernetes Secrets?
The biggest problem with Kubernetes Secrets? They are neither secret nor secure.
When users want to store sensitive data like authentication tokens, Secure Shell Protocol (SSH) keys, passwords and more in Kubernetes, they use the open-source platform’s Secret object. Unfortunately, Kubernetes does little to secure that information once it’s stored in a Secret object. In fact, data stored within a Secret object are merely encoded in Base64 — which is simple to decode and encode — and are not encrypted at all.
An alternative approach might be to add a static encryption key in the encryption configuration file. But since this file is typically hosted on control plane nodes, anyone with access to those nodes can decrypt all secrets.
So how do we actually secure sensitive information in Kubernetes?
We use a key management service (KMS) provider like HashiCorp Vault.
Overview of HashiCorp Vault
HashiCorp Vault has a variety of security applications, but it’s most commonly used for:
- Identity-based access: Using role-based access control (RBAC), Vault grants or restricts access to Secrets creation or other users’ access. Vault also enables the generation of temporary Secrets to provide machine or user access to Secrets or servers on a limited, as-needed basis, ensuring that there aren’t any needlessly permissive access privileges available for exploitation.
- Encryption: Vault offers both encryption in transit and at rest, which is one of the main services we’re interested in when it comes to protecting Kubernetes Secrets. In transit, Vault uses TLS encryption, while at rest, it uses AES 256-bit CBC encryption.
- Secrets management: The main Vault use case with relevance to Kubernetes is its Secrets management capabilities. Vault can store API keys, database credentials, environmental variables and more.
While the encryption and Secrets management features that Vault provides its users are attractive to a Kubernetes use case, there’s still an issue with how HashiCorp Vault works with Kubernetes Secrets. In fact, this challenge is common to most, if not all, KMS services: They aren’t Kubernetes-native.
The Challenge of Kubernetes Secrets Management
By default, Kubernetes stores Secrets objects in the etcd database. KMS service providers like Vault, however, tend to store Secrets externally in their own systems by default.
From the perspective of a third-party KMS vendor, this makes sense. By storing Secrets outside of the Kubernetes etcd database, they can do more with those Secrets and build a product roadmap around the more comprehensive Secrets management capabilities that external storage enables.
This is great for the vendor — it forces developers and infrastructure managers to spend more time with their tool, they become stickier with their customers and more users are forced to develop competency in all the features of their solution. There’s nothing wrong with becoming proficient in a great solution, but most developers and infrastructure managers would prefer to limit the number of tools they use.
Ideally, users could conduct the infrastructure and development-related tasks, such as testing, data protection, ensuring failover and availability, and so on, in the same place they conduct security-related tasks, like Secrets management — that is to say, in Kubernetes.
Keeping as much of the workflow within Kubernetes as possible enables efficiency, increases the number of steps that can be automated and limits the burden of additional training and maintenance. Crucially, Kubernetes also functions the same regardless of which cloud provider(s) you use. In contrast, managing secrets within HashiCorp Vault works differently for each cloud provider, requiring cloud-specific skills and increasing complexity. And when it comes to security, a simple workflow creates much less risk than a complicated one.
Staying within Kubernetes allows you to become agnostic with regard to public cloud providers. Kubernetes is the de facto cloud operating system and can be used as a standard to build modern applications and infrastructure regardless of the cloud that is used, using the same technologies and patterns across all of them. As a result, you can focus on Kubernetes skills, and the cloud-specific skills required are limited. This allows customers to build a strong multicloud strategy.
How to Run Vault on Kubernetes with Kube-Native Secrets Management
Fortunately, Vault provides us with a way to at least get started on a Kube-native approach to Secrets management.
Vault has a variety of different Secrets engines. Essentially, these are different approaches to storing, generating and encrypting data. These engines might focus on connecting to specific services like Amazon Web Services or Google Cloud, simply storing and reading data, providing specific forms of encryption and more.
Our interest lies in the Transit Secrets engine.
Transit functions as an encryption-as-a-service Secrets engine — it doesn’t store the data sent to its engine but simply encrypts or decrypts data in transit.
This allows us to handle Secrets in a Kube-native way. Because Kubernetes wants to and is designed around storing data in etcd, all we need to do is request encryption services from Vault via the Transit Secrets engine any time there’s a need for Kubernetes Secrets management.
Creating Secrets with Vault in a Kube-Native Way
Vault wasn’t designed for Kube-native applications, and Kubernetes only provides a high-level, generic API that offloads the encryption/decryption tasks to an external KMS provider. Thus, DevOps professionals need to do some work in order to integrate the two solutions. To speed that integration up as much as possible, Ondat is sponsoring an open source project called Trousseau.
Trousseau acts as a proxy between Kubernetes and Vault (or any other KMS provider). Using Trousseau, users and workloads can manage Kubernetes Secrets using the native Kubernetes API and kubectl CLI to access encryption-as-a-service from a KMS provider. In the specific use case we’re discussing today, Trousseau would enable a user or workload to use the kubectl CLI to request encryption from Vault’s Transit Secrets engine.
Trousseau works because Kubernetes already offers a plugin to support KMS providers. Implementing Trousseau with this plugin enables users to rapidly connect to any KMS provider and encrypt Secrets on the fly using the Kube-native tools and practices that your team is already proficient in.
Once you create a Secret in Kubernetes, the Kube API calls Trousseau, which then sends the encryption request to a KMS provider (Vault in this instance). Then the KMS provider performs the encryption without storing the associated data. Instead, it sends it back to Trousseau, which then sends the data to Kube API, which stores the encrypted resource in etcd in turn.
The result is a faster time to market, greater security and reduced cost associated with training your team on new tools and practices. You can learn more about the Trousseau project here, as well as a hands-on lab that walks you through setting up Trousseau with Vault and Kubernetes, and receive information on how to join the open source project.
