GoDaddy’s Project to Secure, Rotate Kubernetes Secrets

One of the key advantages of GoDaddy's Kubernetes External Secrets is that it constantly synchronizes the local secrets with the cloud-based secret store.

Jul 16th, 2021 8:42am by Janakiram MSV

Featued image for: GoDaddy’s Project to Secure, Rotate Kubernetes Secrets

Feature image via Unsplash.

Kubernetes has a simple and flexible mechanism to store and retrieve sensitive data through the concept of secrets. Operators can easily create and store secrets such as database passwords within a Kubernetes namespace. Developers writing applications can retrieve secrets through the environment variables or files stored in a well-known location. Access to secrets can be restricted to authorized users and applications through Kubernetes role-based access control (RBAC).

But while Kubernetes secrets provide simplicity and flexibility, they also have some limitations that may turn into DevOps’ nightmare.

Firstly, Kubernetes secrets are not really secrets. They are just base64 encoded strings that can be easily decoded by any user or a pod with access to secrets. If the Kubernetes control plane is configured to support encrypted secrets, you may be able to secure them. But, the majority of the managed Kubernetes services don’t have encryption enabled for secrets.

Secondly, Kubernetes secrets cannot be populated or rotated from a single source of truth. With the rise of Kubernetes at the edge, customers are creating and deploying tens of thousands of clusters. While GitOps comes to the rescue of deployments at scale for the edge clusters, it doesn’t address the problem of centralizing secret management. It is not a good idea to store secrets in static YAML files in a Git repository.

Finally, Kubernetes secrets don’t take advantage of cloud-based secret stores and key management services offering enterprise-grade protection. Most of the managed secret stores in the public cloud have automated rotation policies and robust lifecycle management of secrets. But it’s extremely challenging to connect the dots between the cloud-based secret management services and Kubernetes secrets.

Overview of Kubernetes External Secrets

GoDaddy, one of the leading web-hosting companies, open-sourced an internal project called Kubernetes External Secrets. It precisely addresses the challenges discussed above.

GoDaddy extensively relies on Amazon Web Services‘ EKS for running their Kubernetes infrastructure. The engineering team at GoDaddy realized that there is no integration between EKS and other managed services like Amazon Secrets Manager and AWS Systems Manager. To bridge the gap between the two, they built a Kubernetes custom controller and a custom resource definition called External Secrets.

The Kubernetes External Secrets project initially focused on enabling EKS clusters to synchronize Kubernetes secrets with AWS Secrets Manager. This was done by taking advantage of the IAM role integration with Kubernetes service accounts. When a Kubernetes service account is associated with an AWS IAM role, the pods using the same service account assume the role. This is similar to how EC2 instances assume the IAM role to gain access to AWS-managed services.

But soon, the scope of the Kubernetes External Secrets project has been expanded to support non-EKS clusters and Kubernetes clusters running outside of AWS. The controller can be configured with the typical approach of passing the AWS access key and secret key to invoke AWS Secrets Manager API.

Currently, the Kubernetes External Secrets project supports various secret manager backends running in mainstream public cloud platforms. You can synchronize Kubernetes secrets with AWS Secrets Manager, AWS Systems Manager Agent, AKeyless Vault, Azure Key Vault, Google Cloud Secret Manager, HashiCorp Vault, and more.

The Architecture of Kubernetes External Secrets

When you install Kubernetes External Secrets in a cluster, it deploys a CRD called externalsecrets.kubernetes-client.io, defining the specification for the external secrets custom resource.

It also creates a pod that runs the custom controller.

The custom controller has the AWS credentials passed as environment variables to access the AWS Secrets Manager.

Below is the high-level architecture of GoDaddy’s Kubernetes External Secrets controller:

The custom controller creates a corresponding Kubernetes secret for every External Secret object that the pods can access.

In the below screenshot, you can see the dbsecret External Secret object translated to the Kubernetes secret.

One of the key advantages of Kubernetes External Secrets is that it constantly synchronizes the local secrets with the cloud-based secret store. The last sync status in the above screenshot shows that the secret was refreshed just a few seconds ago. This makes it possible to enable secret rotation in the cloud and automatically propagate it to the edge. The polling interval is a configurable parameter of the controller which can be easily adjusted to meet your requirements.

GoDaddy’s Kubernetes External Secret is an ideal tool for managing secrets at scale. Its GitOps-friendly architecture enables creating, managing, and rotating secrets at scale in multicluster edge deployments.

In the next article, I will walk you through the steps of configuring and deploying Kubernetes External Secrets backed by AWS Secrets Manager. Check back Monday for the second part of this tutorial!