Managing Kubernetes Secrets with AWS Secrets Manager

In the last part of this series, I introduced GoDaddy’s Kubernetes External Secrets Manager. In this installment, we will leverage it to configure secrets backed by Amazon Web Services‘ Secrets Manager.
GoDaddy extensively relies on Amazon Web Services‘ EKS for running their Kubernetes infrastructure. The engineering team at GoDaddy realized that there is no integration between EKS and other managed services like Amazon Secrets Manager and AWS Systems Manager. To bridge the gap between the two, they built a Kubernetes custom controller and a custom resource definition called External Secrets, which can manage and rotate the keys originating in AWS Secrets Manager for any Kubernetes deployment.
Let’s see Kubernetes External Secrets in action.
Prerequisites:
1. Active AWS Subscription
2. AWS CLI
3. Kubernetes Cluster
We will start by creating an AWS Identity Access Management (IAM) Policy and an AWS IAM User with just enough permissions to read the secrets from AWS Secrets Store. In the next step, we will deploy the Kubernetes External Secrets Manager Helm Chart associated with the IAM User credentials. After that, we create a secret in the AWS Secrets Store and pointing it to the Kubernetes External Secret created in the local cluster. Finally, we update the cloud-based secret and verify if it’s refreshed in the Kubernetes cluster.
Step 1: Create an AWS IAM Policy and IAM User to Access Secrets Store
Let’s start by defining the IAM Policy needed to access the secrets. Create a JSON file with the below content and save it in extsecpol.json
file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetRandomPassword", "secretsmanager:GetResourcePolicy", "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret", "secretsmanager:ListSecretVersionIds", "secretsmanager:ListSecrets" ], "Resource": "*" } ] } |
Define the environment variables and proceed with the creation of IAM Policy and Role.
1 2 |
POLICYNAME="ExternalSecurityPolicy" IAMUSERNAME="SecretReader" |
1 2 3 |
aws iam create-policy \ --policy-name $POLICYNAME \ --policy-document file://extsecpol.json |
Retrieve and store the Amazon Resource Name (ARN) for the policy in an environment variable.
1 |
export POLICYARN=$(aws iam list-policies --query 'Policies[?PolicyName==`ExternalSecurityPolicy`].{ARN:Arn}' --output text) |
Make sure you replace ExternalSecurityPolicy
with the correct policy name.
Create the IAM User based on the above policy.
1 2 3 |
aws iam create-user --user-name $IAMUSERNAME aws iam attach-user-policy --user-name $IAMUSERNAME --policy-arn $POLICYARN aws iam create-access-key --user-name $IAMUSERNAME |
From the output of the last command, retrieve the Access Key and Secret Key associated with the user.
Step 2: Install Kubernetes External Secrets Helm Chart
Before we deploy the Kubernetes External Secrets custom controller, we need to register the AWS AccessKey and SecretAccessKey as Kubernetes secrets in the same namespace where the custom controller runs.
Let’s start by creating the namespace.
1 |
kubectl create namespace external-secrets |
Encode AccessKey and SecretAccessKey into base64 and turn them into a Kubernetes secret.
1 2 |
echo -n "AKIATWRVOGK5V3ZITMWO" | base64 echo -n "6nGIROhIkZGFFvvhkTFWefX5ONFCZyvZZdTIDllR" | base64 |
Create a file named aws-secrets.yaml
with the base64-encoded values and apply it with kubectl
command.
1 2 3 4 5 6 7 8 |
apiVersion: v1 kind: Secret metadata: name: aws-credentials type: Opaque data: id: QUtJQVRXUlZPR0s1VjNaSVRNV08= key: Nm5HSVJPaElrWkdGRnZ2aGtURldlZlg1T05GQ1p5dlpaZFRJRGxsUg== |
1 |
kubectl apply -f aws-secrets.yaml -n external-secrets |
Let’s associate the secret with the Helm Chart by defining the below values.yaml
file:
1 2 3 4 5 6 7 8 9 10 11 |
env: AWS_REGION: ap-south-1 AWS_DEFAULT_REGION: ap-south-1 envVarsFromSecret: AWS_ACCESS_KEY_ID: secretKeyRef: "aws-credentials" key: "id" AWS_SECRET_ACCESS_KEY: secretKeyRef: "aws-credentials" key: "key" |
Replace the AWS_REGION with the region where you created the secret.
Now, we are ready to install the Kubernetes External Secrets Helm Chart.
1 2 |
helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/ helm repo update |
1 2 3 4 5 |
helm install \ --generate-name \ --namespace external-secrets \ external-secrets/kubernetes-external-secrets \ --values values.yaml |
Verify the deployment and ensure there are no errors.
1 |
kubectl get pods -n external-secrets |
Step 3: Create a Secret in AWS Secrets Manager
With the Kubernetes cluster connected to the AWS Secrets Manager through the External Secrets controller, we are ready to create secrets in the cloud and propagate them to the local cluster.
1 2 3 4 |
aws secretsmanager create-secret \ --region ap-south-1 \ --name edgesecrets/dbcred \ --secret-string '{"username":"admin","password":"password@1234"}' |
In the next step, we will create an External Secret pointed to the secret created in the AWS Secrets Manager.
Step 4: Create an External Secret Resource in Kubernetes
Let’s create an External Secret custom resource called dbcred
associated with the cloud-based secret created in the previous step in the default namespace.
1 2 3 4 5 6 7 8 9 |
apiVersion: "kubernetes-client.io/v1" kind: ExternalSecret metadata: name: dbcred spec: backendType: secretsManager region: ap-south-1 dataFrom: - edgesecrets/dbcred |
1 |
kubectl create -f dbcred-external.yaml |
Verify that the External Secret and the Kubernetes secret are created successfully.
1 |
kubectl get es |
1 |
kubectl get secrets |
Let’s retrieve the values of the secret by decoding the base64 values.
1 |
kubectl get secret dbcred -o yaml |
1 |
echo " YWRtaW4=" | base64 -d |
1 |
echo "cGFzc3dvcmRAMTIzNA==" | base64 -d |
Access the AWS Management Console to retrieve the secret values.
As you can see, the same values showed in the AWS Console are available in the Kubernetes cluster.
Step 5: Rotating/Refreshing the Secrets
Let’s now modify the secret stored on AWS Secrets Store to see if it automatically gets propagated to the Kubernetes cluster.
Run the below command to change the password to a new value:
1 2 3 4 |
aws secretsmanager update-secret \ --region ap-south-1 \ --secret-id edgesecrets/dbcred \ --secret-string '{"username":"admin","password":"password@0000"}' |
Retrieve the password value from the Kubernetes cluster.
1 |
kubectl get secret dbcred -o yaml |
1 |
echo "cGFzc3dvcmRAMDAwMA==" | base64 -d |
As we can see, the updated password instantly becomes available in the Kubernetes cluster.
Step 6: Cleaning up Resources
Delete Kubernetes resources:
1 2 3 4 |
kubectl delete -f dbcred-external.yaml helm ls -n external-secrets helm delete RELEASE_NAME kubectl delete -f aws-secrets.yaml -n external-secrets |
Delete AWS IAM Policy:
1 2 3 |
IAMUSERNAME=SecretReader export POLICYARN=$(aws iam list-policies --query 'Policies[?PolicyName==`ExternalSecurityPolicy`].{ARN:Arn}' --output text) aws iam detach-user-policy --user-name $IAMUSERNAME --policy-arn $POLICYARN |
Delete AWS IAM User:
1 2 3 4 5 |
export ACCESSKEYID=$(aws iam list-access-keys --user-name SecretReader --query 'AccessKeyMetadata[0].AccessKeyId' --output text) aws iam delete-access-key --user-name $IAMUSERNAME --access-key-id $ACCESSKEYID aws iam delete-user --user-name $IAMUSERNAME aws iam delete-policy --policy-arn $POLICYARN |
Delete AWS Secret Manager secret:
1 |
aws secretsmanager delete-secret --secret-id edgesecrets/dbcred |