Managing Kubernetes Secrets with AWS Secrets Manager

In the last part of this series, I introduced GoDaddy’s Kubernetes External Secrets Manager. In this installment, we will leverage it to configure secrets backed by Amazon Web Services‘ Secrets Manager.

GoDaddy extensively relies on Amazon Web Services‘ EKS for running their Kubernetes infrastructure. The engineering team at GoDaddy realized that there is no integration between EKS and other managed services like Amazon Secrets Manager and AWS Systems Manager. To bridge the gap between the two, they built a Kubernetes custom controller and a custom resource definition called External Secrets, which can manage and rotate the keys originating in AWS Secrets Manager for any Kubernetes deployment.

Let’s see Kubernetes External Secrets in action.

Prerequisites:

1. Active AWS Subscription
2. AWS CLI
3. ​​Kubernetes Cluster

We will start by creating an AWS Identity Access Management (IAM) Policy and an AWS IAM User with just enough permissions to read the secrets from AWS Secrets Store. In the next step, we will deploy the Kubernetes External Secrets Manager Helm Chart associated with the IAM User credentials. After that, we create a secret in the AWS Secrets Store and pointing it to the Kubernetes External Secret created in the local cluster. Finally, we update the cloud-based secret and verify if it’s refreshed in the Kubernetes cluster.

Step 1: Create an AWS IAM Policy and IAM User to Access Secrets Store

Let’s start by defining the IAM Policy needed to access the secrets. Create a JSON file with the below content and save it in extsecpol.json file.

Define the environment variables and proceed with the creation of IAM Policy and Role.

Retrieve and store the Amazon Resource Name (ARN) for the policy in an environment variable.

Make sure you replace ExternalSecurityPolicy with the correct policy name.

Create the IAM User based on the above policy.

From the output of the last command, retrieve the Access Key and Secret Key associated with the user.

Step 2: Install Kubernetes External Secrets Helm Chart

Before we deploy the Kubernetes External Secrets custom controller, we need to register the AWS AccessKey and SecretAccessKey as Kubernetes secrets in the same namespace where the custom controller runs.

Let’s start by creating the namespace.

Encode AccessKey and SecretAccessKey into base64 and turn them into a Kubernetes secret.

Create a file named aws-secrets.yaml with the base64-encoded values and apply it with kubectl command.

Let’s associate the secret with the Helm Chart by defining the below values.yaml file:

Replace the AWS_REGION with the region where you created the secret.

Now, we are ready to install the Kubernetes External Secrets Helm Chart.

Verify the deployment and ensure there are no errors.

Step 3: Create a Secret in AWS Secrets Manager

With the Kubernetes cluster connected to the AWS Secrets Manager through the External Secrets controller, we are ready to create secrets in the cloud and propagate them to the local cluster.

In the next step, we will create an External Secret pointed to the secret created in the AWS Secrets Manager.

Step 4: Create an External Secret Resource in Kubernetes

Let’s create an External Secret custom resource called dbcred associated with the cloud-based secret created in the previous step in the default namespace.

Verify that the External Secret and the Kubernetes secret are created successfully.

Let’s retrieve the values of the secret by decoding the base64 values.

Access the AWS Management Console to retrieve the secret values.

As you can see, the same values showed in the AWS Console are available in the Kubernetes cluster.

Step 5: Rotating/Refreshing the Secrets

Let’s now modify the secret stored on AWS Secrets Store to see if it automatically gets propagated to the Kubernetes cluster.

Run the below command to change the password to a new value:

Retrieve the password value from the Kubernetes cluster.

As we can see, the updated password instantly becomes available in the Kubernetes cluster.

Step 6: Cleaning up Resources

Delete Kubernetes resources:

Delete AWS IAM Policy:

Delete AWS IAM User:

Delete AWS Secret Manager secret: