Using the Kyverno CLI to Write Policy Test Cases
Kubernetes has become one of the most popular container orchestrators for deploying applications at scale. And building the cluster securely has always been a matter of concern. Many tools can scan your cluster and list its security threats. But implementing them can be challenging as one has to write and manage custom webhooks for each of them.
So, policy as a code has been introduced to help Kuberentes admins and make their life easier. Policy means a set of rules that must be met to impose security and govern IT operations easily. Policy as a code is an approach for policy management using high-level languages such as YAML or Rego.
There are different policy engines, such as Kyverno and OPA, to impose policies as code to secure the Kubernetes cluster.
In this blog, we will learn more about Kyverno, its policies, its use in the Kubernetes cluster and how to set up a Kyverno project locally. Along with this, we’ll look at how to work with the Kyverno command line interface (CLI), write test cases for validating and mutating policies, and test them with the Kyverno CLI, which has also been my Linux Foundation mentorship work.
Let’s first understand what the admission controller in Kubernetes is as Kyverno uses it.
Admission Controller in Kubernetes
An admission controller in Kubernetes is a piece of code that intercepts the incoming object request to the Kuberentes API server before persisting it into the ETCD. But this is done only after the request is authenticated and authorized. These controllers validate or mutate the incoming request based on the validating and mutating admission webhooks, generally known as dynamic admission controllers.
The following diagram illustrates its working inside the Kubernetes cluster.
Kyverno inside the cluster works in place of these webhooks, making the work easier through policies.
What Is Kyverno?
Kyverno, a Greek word for “govern,” is a policy engine natively designed for Kubernetes. Currently, it is an incubating project under the Cloud Native Computing Foundation (CNCF). Kyverno policies are managed as Kubernetes resources and are written in YAML. So, no new language is required to write Kyverno policies.
Features of Kyverno
Various features of Kyverno make it unique from other policy engines, such as:
- Policies are managed as Kubernetes resources
- Policies are manageable with kubectl, git and Kustomize tools
- Validate, mutate, generate or even remove any resources
- Helps in container image signing and verification for software supply chain security
- Match resources using label selectors and wildcards
- Enable background scans on existing Kubernetes resources to ensure best practices
- Block resources and report policy violations
- Generate policy reports
- Test policies and validate resources with Kyverno CLI in CI/CD pipeline before using them in a cluster
How Kyverno Works
Kyverno is used as a dynamic admission controller inside the Kuberentes cluster. It receives validating and mutating admission webhook HTTP callbacks from the kube-apiserver and applies matching policies to return results that enforce admission policies or reject requests.
Figure reference: https://kyverno.io/docs/introduction/#how-kyverno-works
Kyverno policies can match resources with the help of resource kind, name, label selectors, user, and much more. Policy enforcement is recorded through Kubernetes events, and it also reports violations of existing resources through background scans.
Policy Types and Their Behaviors
Two types of policies are permitted with Kyverno:
- ClusterPolicy: The policy scope is cluster-wide.
- Policy: The policy scope is limited to the namespace in which it is applied.
Figure reference: https://kyverno.io/docs/kyverno-policies/
Different Behaviors of Policies
The different behaviors found in Kyverno policies for resources are:
- Validate: This uses overlay-style syntax with pattern-matching support and conditional (if-then-else) processing. If the Kubernetes resource matches the pattern specified, then the resource is allowed to be created, otherwise, the creation will be blocked.
- Mutate: This uses RFC 6902 JSONPatch format, which is used to modify the matching resources.
- Generate: This is used when there is a need to create supporting resources based on a new resource.
- Verify Images: With the help of Sigstore’s sub-project Cosign, container image signatures are checked.
Kyverno in the Kubernetes Cluster
Kyverno Installation
| helm repo add kyverno https://kyverno.github.io/kyverno/ helm repo update helm install kyverno kyverno/kyverno -n kyverno –create-namespace –set replicaCount=1 |
- Verify the pods in the Kyverno namespace are running.
| kubectl get pods -n kyverno |
For now, we will see Kyverno policies with validate and mutate behavior on Kubernetes resources.
Validating Policy in Kyverno
- To ensure pods are allowed to mount
hostPath volumes inreadOnlymode. This validating policy checks all the containers of pods for any 1hostPath1 volumes and ensures that all of them are only inreadOnlymode.
Policy reference: https://kyverno.io/policies/other/ensure_readonly_hostpath/ensure_readonly_hostpath/
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: ensure-readonly-hostpath annotations: policies.kyverno.io/title: Ensure Read Only hostPath policies.kyverno.io/category: Other policies.kyverno.io/severity: medium policies.kyverno.io/minversion: 1.6.0 kyverno.io/kyverno-version: 1.6.2 kyverno.io/kubernetes-version: "1.23" policies.kyverno.io/subject: Pod policies.kyverno.io/description: >- Pods which are allowed to mount hostPath volumes in read/write mode pose a security risk even if confined to a "safe" file system on the host and may escape those confines (see https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts). The only true way to ensure safety is to enforce that all Pods mounting hostPath volumes do so in read only mode. This policy checks all containers for any hostPath volumes and ensures they are explicitly mounted in readOnly mode. spec: background: false validationFailureAction: Enforce rules: - name: ensure-hostpaths-readonly match: any: - resources: kinds: - Pod preconditions: all: - key: "{{ request.operation || 'BACKGROUND' }}" operator: AnyIn value: - CREATE - UPDATE validate: message: All hostPath volumes must be mounted as readOnly. foreach: - list: "request.object.spec.volumes[?hostPath][]" deny: conditions: any: - key: "{{ request.object.spec.[containers, initContainers, ephemeralContainers][].volumeMounts[?name == '{{element.name}}'][] | length(@) }}" operator: NotEquals value: "{{ request.object.spec.[containers, initContainers, ephemeralContainers][].volumeMounts[?name == '{{element.name}}' && readOnly] [] | length(@) }}" |
When the policy attribute validationFailureAction is set to “Audit” (default) mode, it will report policy violations by resources but will not block them from being created. On the other hand, “Enforce” mode will block them as well.
| kubectl apply -f ensure-readonly-hostpath.yaml kubectl get clusterpolicies |
- Create new pod resources with
hostPathvolume to test policy.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
apiVersion: v1 kind: Pod metadata: name: nginx-without-readonly spec: containers: - image: nginx:latest name: nginx volumeMounts: - mountPath: /data name: nginx-volume volumes: - name: nginx-volume hostPath: path: /data --- apiVersion: v1 kind: Pod metadata: name: nginx-with-readonly spec: containers: - image: nginx:latest name: nginx volumeMounts: - mountPath: /data name: nginx-volume readOnly: true volumes: - name: nginx-volume hostPath: path: /data |
On creating the above pod resources, one which has hostPath mounted as readOnly will be created and the other one will be blocked from being created.
Mutating Policy in Kyverno
- It is always a good practice to work with fully resolved container images that have digest in them rather than the tag. This mutate policy changes the pod container images and sets them as fully resolved.
# Policy reference: https://kyverno.io/policies/other/resolve_image_to_digest/resolve-image-to-digest/
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: resolve-image-to-digest annotations: policies.kyverno.io/title: Resolve Image to Digest policies.kyverno.io/category: Other policies.kyverno.io/severity: medium kyverno.io/kyverno-version: 1.6.0 policies.kyverno.io/minversion: 1.6.0 kyverno.io/kubernetes-version: "1.23" policies.kyverno.io/subject: Pod policies.kyverno.io/description: >- Image tags are mutable and the change of an image can result in the same tag. This policy resolves the image digest of each image in a container and replaces the image with the fully resolved reference which includes the digest rather than tag. spec: background: false rules: - name: resolve-to-digest match: any: - resources: kinds: - Pod preconditions: all: - key: "{{request.operation || 'BACKGROUND'}}" operator: NotEquals value: DELETE mutate: foreach: - list: "request.object.spec.containers" context: - name: resolvedRef imageRegistry: reference: "{{ element.image }}" jmesPath: "resolvedImage" patchStrategicMerge: spec: containers: - name: "{{ element.name }}" image: "{{ resolvedRef }}" kubectl apply -f resolve-image-to-digest.yaml kubectl get clusterpolicies |
- Create new pod resource to test the policy
|
1 2 3 4 5 6 7 8 |
apiVersion: v1 kind: Pod metadata: name: nginx-app spec: containers: - image: nginx:latest name: nginx |
From the above output, it can be clearly seen that the NGINX image has been fully resolved along with the image digest.
Setting Up Kyverno Locally
To set up the Kyverno project locally, you must have Golang installed on your machine. Ensure that the GOPATH environment variable is set up correctly.
- Now clone the Kyverno repository locally on your system under a subdirectory of your
$GOPATH location, as shown below.
| git clone https://github.com/kyverno/kyverno.git $GOPATH/src/github.com/kyverno/kyverno |
- Similarly, you can clone the policies repository by following the above path.
| git clone https://github.com/kyverno/policies $GOPATH/src/github.com/kyverno/policies |
You can also check out the Kyverno project wiki for further guidance You can also create a local cluster with Kind/Minikube and build a local Kyverno image and use them in your local cluster. Find out more from these training videos.
- Install the Kyverno CLI. One can find more information about different versions of it on the release page.
| wget https://github.com/kyverno/kyverno/releases/download/v1.9.2/kyverno-cli_v1.9.2_linux_x86_64.tar.gz
tar -xvzf kyverno-cli_v1.9.2_linux_x86_64.tar.gz mv kyverno /usr/local/bin |
- Verify the Kyverno CLI installation by checking its version.
| kyverno version |
What Is Kyverno CLI?
The Kyverno CLI is designed to apply and test policies outside the cluster. This is mainly used to validate and test the policy behavior on resources before adding them to the cluster. This CLI can be used with kubectl as a plugin, CI/CD pipelines or as a standalone.
Different Kyverno CLI Commands
Different subcommands can be used with the Kyverno CLI, such as:
Kyverno Apply Command
The kyverno apply command is used to try out the policies with given resources. For validating policies, it gives out the result as pass/fail/skip depending on the resource given against the policy. In the case of mutate policies, it generates the mutated resource as an output.
The syntax of the command is as follows:
| kyverno apply /path/to/policy.yaml –resource /path/to/resource.yaml -f /path/to/values.yaml |
The -f flag is used to pass variables that are needed to test the policy against the resources. Find out more about it here.
Kyverno Test Command
The kyverno test command is used to test the resources against policies for desired results. For this, a separate (standard) kyverno-test.yaml file is created to mention the tests.
The syntax of the command is as follows:
| kyverno test /path/to/testyamlfile |
The -f flag is used to set a custom file name, which includes test cases. By default, it will search for a file called kyverno-test.yaml. Find out more about it here.
Now that we learned about the Kyverno CLI, we can discuss how to write the test cases for resources to be tested against the policies.
Writing Test Cases for Kyverno Policies and Testing Them with Kyverno CLI
To write test cases one by one, we will use the same validating and mutating policies we used above.
Tests for Validating Policy
- This validating policy will work on pod resources that have a precondition and rule for matching
hostPathvolume. This precondition is used to have more control over rules by defining variables that will be defined in thevalues.yamlfile.
It’s vital to have negative and positive cases to ensure when the policy is being passed and when it is being failed.
- Below is the
values.yamlandkyverno-test.yamlfile for the validating policy.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# values.yaml policies: - name: ensure-readonly-hostpath resources: - name: good-pod-01 values: request.operation: UPDATE # kyverno-test.yaml name: ensure-readonly-hostpath policies: - ensure-readonly-hostpath.yaml resources: - good-pod-01.yaml - bad-pod-01.yaml variables: values.yaml results: - policy: ensure-readonly-hostpath rule: ensure-hostpaths-readonly resource: bad-pod-01 kind: Pod result: fail - policy: ensure-readonly-hostpath rule: ensure-hostpaths-readonly resource: good-pod-01 kind: Pod result: pass |
- Let’s check the policy with these test cases on Kyverno locally with the Kyverno CLI command.
| kyverno apply ensure-readonly-hostpath.yaml -r good-pod-01.yaml -f values.yaml |
| kyverno test . |
Tests for Mutating Policy
- This mutating policy will work on a pod resource with a mutate rule to change image tag to digest and a precondition with it. The
kyverno applycommand will generate a mutate (patchedResource.yaml) file for the resource on which the policy is applied.
- Below is the
values.yaml,kyverno-test.yamlandpatchedResource.yamlfile for the mutating policy.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
# values.yaml policies: - name: resolve-image-to-digest rules: - name: resolve-to-digest values: resolvedRef: "busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47" resources: - name: busybox values: request.operation: UPDATE # kyverno-test.yaml name: resolve-image-to-digest policies: - resolve-image-to-digest.yaml resources: - pod.yaml variables: values.yaml results: - policy: resolve-image-to-digest rule: resolve-to-digest resource: busybox patchedResource: patchedResource.yaml kind: Pod result: pass # patchedResource.yaml apiVersion: v1 kind: Pod metadata: name: busybox spec: containers: - name: busybox image: busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 |
- Let’s check the policy with these test cases on Kyverno locally with the Kyverno CLI command.
| kyverno apply resolve-image-to-digest.yaml -r pod.yaml -f values.yaml |
| kyverno test . |
So, this is how to write test cases for Kyverno policies and test them locally.
