Using the Kyverno CLI to Write Policy Test Cases

A look at Kyverno, its policies, its use in the Kubernetes cluster and how to set up a Kyverno project locally.

May 9th, 2023 6:14am by Oshi Gupta

Featued image for: Using the Kyverno CLI to Write Policy Test Cases

Kubernetes has become one of the most popular container orchestrators for deploying applications at scale. And building the cluster securely has always been a matter of concern. Many tools can scan your cluster and list its security threats. But implementing them can be challenging as one has to write and manage custom webhooks for each of them.

So, policy as a code has been introduced to help Kuberentes admins and make their life easier. Policy means a set of rules that must be met to impose security and govern IT operations easily. Policy as a code is an approach for policy management using high-level languages such as YAML or Rego.

There are different policy engines, such as Kyverno and OPA, to impose policies as code to secure the Kubernetes cluster.

In this blog, we will learn more about Kyverno, its policies, its use in the Kubernetes cluster and how to set up a Kyverno project locally. Along with this, we’ll look at how to work with the Kyverno command line interface (CLI), write test cases for validating and mutating policies, and test them with the Kyverno CLI, which has also been my Linux Foundation mentorship work.

Let’s first understand what the admission controller in Kubernetes is as Kyverno uses it.

Admission Controller in Kubernetes

An admission controller in Kubernetes is a piece of code that intercepts the incoming object request to the Kuberentes API server before persisting it into the ETCD. But this is done only after the request is authenticated and authorized. These controllers validate or mutate the incoming request based on the validating and mutating admission webhooks, generally known as dynamic admission controllers.

The following diagram illustrates its working inside the Kubernetes cluster.

Kyverno inside the cluster works in place of these webhooks, making the work easier through policies.

What Is Kyverno?

Kyverno, a Greek word for “govern,” is a policy engine natively designed for Kubernetes. Currently, it is an incubating project under the Cloud Native Computing Foundation (CNCF). Kyverno policies are managed as Kubernetes resources and are written in YAML. So, no new language is required to write Kyverno policies.

Features of Kyverno

Various features of Kyverno make it unique from other policy engines, such as:

Policies are managed as Kubernetes resources
Policies are manageable with kubectl, git and Kustomize tools
Validate, mutate, generate or even remove any resources
Helps in container image signing and verification for software supply chain security
Match resources using label selectors and wildcards
Enable background scans on existing Kubernetes resources to ensure best practices
Block resources and report policy violations
Generate policy reports
Test policies and validate resources with Kyverno CLI in CI/CD pipeline before using them in a cluster

How Kyverno Works

Kyverno is used as a dynamic admission controller inside the Kuberentes cluster. It receives validating and mutating admission webhook HTTP callbacks from the kube-apiserver and applies matching policies to return results that enforce admission policies or reject requests.

Figure reference: https://kyverno.io/docs/introduction/#how-kyverno-works

Kyverno policies can match resources with the help of resource kind, name, label selectors, user, and much more. Policy enforcement is recorded through Kubernetes events, and it also reports violations of existing resources through background scans.

Policy Types and Their Behaviors

Two types of policies are permitted with Kyverno:

ClusterPolicy: The policy scope is cluster-wide.
Policy: The policy scope is limited to the namespace in which it is applied.

Figure reference: https://kyverno.io/docs/kyverno-policies/

Different Behaviors of Policies

The different behaviors found in Kyverno policies for resources are:

Validate: This uses overlay-style syntax with pattern-matching support and conditional (if-then-else) processing. If the Kubernetes resource matches the pattern specified, then the resource is allowed to be created, otherwise, the creation will be blocked.
Mutate: This uses RFC 6902 JSONPatch format, which is used to modify the matching resources.
Generate: This is used when there is a need to create supporting resources based on a new resource.
Verify Images: With the help of Sigstore’s sub-project Cosign, container image signatures are checked.

Kyverno in the Kubernetes Cluster

Kyverno Installation

- To work with Kyverno policies in the Kubernetes cluster, first, install Kyverno custom resource definitions (CRDs) either via Helm or kubectl.

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno –create-namespace –set replicaCount=1

Verify the pods in the Kyverno namespace are running.

kubectl get pods -n kyverno

For now, we will see Kyverno policies with validate and mutate behavior on Kubernetes resources.

Validating Policy in Kyverno

To ensure pods are allowed to mount hostPath volumes in readOnly mode. This validating policy checks all the containers of pods for any 1hostPath1 volumes and ensures that all of them are only in readOnly mode.

Policy reference: https://kyverno.io/policies/other/ensure_readonly_hostpath/ensure_readonly_hostpath/

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: ensure-readonly-hostpath
  annotations:
    policies.kyverno.io/title: Ensure Read Only hostPath
    policies.kyverno.io/category: Other
    policies.kyverno.io/severity: medium
    policies.kyverno.io/minversion: 1.6.0
    kyverno.io/kyverno-version: 1.6.2
    kyverno.io/kubernetes-version: "1.23"
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Pods which are allowed to mount hostPath volumes in read/write mode pose a security risk
      even if confined to a "safe" file system on the host and may escape those confines (see
      https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts). The only true way
      to ensure safety is to enforce that all Pods mounting hostPath volumes do so in read only
      mode. This policy checks all containers for any hostPath volumes and ensures they are
      explicitly mounted in readOnly mode.      
spec:
  background: false
  validationFailureAction: Enforce
  rules:
  - name: ensure-hostpaths-readonly
    match:
      any:
      - resources:
          kinds:
            - Pod
    preconditions:
      all:
      - key: "{{ request.operation || 'BACKGROUND' }}"
        operator: AnyIn
        value:
        - CREATE
        - UPDATE
    validate:
      message: All hostPath volumes must be mounted as readOnly.
      foreach:
      - list: "request.object.spec.volumes[?hostPath][]"
        deny:
          conditions:
            any:
            - key: "{{ request.object.spec.[containers, initContainers, ephemeralContainers][].volumeMounts[?name == '{{element.name}}'][] | length(@) }}"
              operator: NotEquals
              value: "{{ request.object.spec.[containers, initContainers, ephemeralContainers][].volumeMounts[?name == '{{element.name}}' && readOnly] [] | length(@) }}"

apiVersion: kyverno.io/v1

kind: ClusterPolicy

metadata:

annotations:

policies.kyverno.io/title: Ensure Read Only hostPath

policies.kyverno.io/category: Other

policies.kyverno.io/severity: medium

policies.kyverno.io/minversion: 1.6.0

kyverno.io/kyverno-version: 1.6.2

kyverno.io/kubernetes-version: "1.23"

policies.kyverno.io/subject: Pod

policies.kyverno.io/description: >-

Pods which are allowed to mount hostPath volumes in read/write mode pose a security risk

even if confined to a "safe" file system on the host and may escape those confines (see

https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts). The only true way

to ensure safety is to enforce that all Pods mounting hostPath volumes do so in read only

mode. This policy checks all containers for any hostPath volumes and ensures they are

explicitly mounted in readOnly mode.

spec:

background: false

validationFailureAction: Enforce

rules:

- name: ensure-hostpaths-readonly

match:

any:

- resources:

kinds:

- Pod

preconditions:

all:

- key: "{{ request.operation || 'BACKGROUND' }}"

operator: AnyIn

value:

- CREATE

- UPDATE

validate:

message: All hostPath volumes must be mounted as readOnly.

foreach:

- list: "request.object.spec.volumes[?hostPath][]"

deny:

conditions:

any:

- key: "{{ request.object.spec.[containers, initContainers, ephemeralContainers][].volumeMounts[?name == '{{element.name}}'][] | length(@) }}"

operator: NotEquals

value: "{{ request.object.spec.[containers, initContainers, ephemeralContainers][].volumeMounts[?name == '{{element.name}}' && readOnly] [] | length(@) }}"

When the policy attribute validationFailureAction is set to “Audit” (default) mode, it will report policy violations by resources but will not block them from being created. On the other hand, “Enforce” mode will block them as well.

kubectl apply -f ensure-readonly-hostpath.yaml
kubectl get clusterpolicies

Create new pod resources with hostPath volume to test policy.

apiVersion: v1

kind: Pod

metadata:

spec:

containers:

- image: nginx:latest

volumeMounts:

- mountPath: /data

volumes:

- name: nginx-volume

hostPath:

path: /data

---

apiVersion: v1

kind: Pod

metadata:

spec:

containers:

- image: nginx:latest

volumeMounts:

- mountPath: /data

readOnly: true

volumes:

- name: nginx-volume

hostPath:

path: /data

On creating the above pod resources, one which has hostPath mounted as readOnly will be created and the other one will be blocked from being created.

Mutating Policy in Kyverno

It is always a good practice to work with fully resolved container images that have digest in them rather than the tag. This mutate policy changes the pod container images and sets them as fully resolved.

# Policy reference: https://kyverno.io/policies/other/resolve_image_to_digest/resolve-image-to-digest/

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: resolve-image-to-digest
  annotations:
    policies.kyverno.io/title: Resolve Image to Digest
    policies.kyverno.io/category: Other
    policies.kyverno.io/severity: medium
    kyverno.io/kyverno-version: 1.6.0
    policies.kyverno.io/minversion: 1.6.0
    kyverno.io/kubernetes-version: "1.23"
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Image tags are mutable and the change of an image can result in the same tag.
      This policy resolves the image digest of each image in a container and replaces
      the image with the fully resolved reference which includes the digest rather than tag.      
spec:
  background: false
  rules:
  - name: resolve-to-digest
    match:
      any:
      - resources:
          kinds:
          - Pod
    preconditions:
      all:
      - key: "{{request.operation || 'BACKGROUND'}}"
        operator: NotEquals
        value: DELETE
    mutate:
      foreach:
      - list: "request.object.spec.containers"
        context:
          - name: resolvedRef
            imageRegistry:
              reference: "{{ element.image }}"
              jmesPath: "resolvedImage"
        patchStrategicMerge:
          spec:
            containers:
            - name: "{{ element.name }}"           
              image: "{{ resolvedRef }}"

kubectl apply -f resolve-image-to-digest.yaml
kubectl get clusterpolicies

apiVersion: kyverno.io/v1

kind: ClusterPolicy

metadata:

annotations:

policies.kyverno.io/title: Resolve Image to Digest

policies.kyverno.io/category: Other

policies.kyverno.io/severity: medium

kyverno.io/kyverno-version: 1.6.0

policies.kyverno.io/minversion: 1.6.0

kyverno.io/kubernetes-version: "1.23"

policies.kyverno.io/subject: Pod

policies.kyverno.io/description: >-

Image tags are mutable and the change of an image can result in the same tag.

This policy resolves the image digest of each image in a container and replaces

the image with the fully resolved reference which includes the digest rather than tag.

spec:

background: false

rules:

- name: resolve-to-digest

match:

any:

- resources:

kinds:

- Pod

preconditions:

all:

- key: "{{request.operation || 'BACKGROUND'}}"

operator: NotEquals

value: DELETE

mutate:

foreach:

- list: "request.object.spec.containers"

context:

- name: resolvedRef

imageRegistry:

reference: "{{ element.image }}"

jmesPath: "resolvedImage"

patchStrategicMerge:

spec:

containers:

- name: "{{ element.name }}"

image: "{{ resolvedRef }}"

kubectl apply -f resolve-image-to-digest.yaml

kubectl get clusterpolicies

Create new pod resource to test the policy

apiVersion: v1

kind: Pod

metadata:

spec:

containers:

- image: nginx:latest

From the above output, it can be clearly seen that the NGINX image has been fully resolved along with the image digest.

Setting Up Kyverno Locally

To set up the Kyverno project locally, you must have Golang installed on your machine. Ensure that the GOPATH environment variable is set up correctly.

Now clone the Kyverno repository locally on your system under a subdirectory of your $GOPATH location, as shown below.

git clone https://github.com/kyverno/kyverno.git $GOPATH/src/github.com/kyverno/kyverno

Similarly, you can clone the policies repository by following the above path.

git clone https://github.com/kyverno/policies
$GOPATH/src/github.com/kyverno/policies

You can also check out the Kyverno project wiki for further guidance You can also create a local cluster with Kind/Minikube and build a local Kyverno image and use them in your local cluster. Find out more from these training videos.

Install the Kyverno CLI. One can find more information about different versions of it on the release page.

wget https://github.com/kyverno/kyverno/releases/download/v1.9.2/kyverno-cli_v1.9.2_linux_x86_64.tar.gz

tar -xvzf kyverno-cli_v1.9.2_linux_x86_64.tar.gz

mv kyverno /usr/local/bin

Verify the Kyverno CLI installation by checking its version.

kyverno version

What Is Kyverno CLI?

The Kyverno CLI is designed to apply and test policies outside the cluster. This is mainly used to validate and test the policy behavior on resources before adding them to the cluster. This CLI can be used with kubectl as a plugin, CI/CD pipelines or as a standalone.

Different Kyverno CLI Commands

Different subcommands can be used with the Kyverno CLI, such as:

Kyverno Apply Command

The kyverno apply command is used to try out the policies with given resources. For validating policies, it gives out the result as pass/fail/skip depending on the resource given against the policy. In the case of mutate policies, it generates the mutated resource as an output.

The syntax of the command is as follows:

kyverno apply /path/to/policy.yaml –resource /path/to/resource.yaml -f /path/to/values.yaml

The -f flag is used to pass variables that are needed to test the policy against the resources. Find out more about it here.

Kyverno Test Command

The kyverno test command is used to test the resources against policies for desired results. For this, a separate (standard) kyverno-test.yaml file is created to mention the tests.

The syntax of the command is as follows:

kyverno test /path/to/testyamlfile

The -f flag is used to set a custom file name, which includes test cases. By default, it will search for a file called kyverno-test.yaml. Find out more about it here.

Now that we learned about the Kyverno CLI, we can discuss how to write the test cases for resources to be tested against the policies.

Writing Test Cases for Kyverno Policies and Testing Them with Kyverno CLI

To write test cases one by one, we will use the same validating and mutating policies we used above.

Tests for Validating Policy

This validating policy will work on pod resources that have a precondition and rule for matching hostPath volume. This precondition is used to have more control over rules by defining variables that will be defined in the values.yaml file.

It’s vital to have negative and positive cases to ensure when the policy is being passed and when it is being failed.

Below is the values.yaml andkyverno-test.yaml file for the validating policy.

# values.yaml

policies:

- name: ensure-readonly-hostpath

resources:

- name: good-pod-01

values:

request.operation: UPDATE

# kyverno-test.yaml

policies:

- ensure-readonly-hostpath.yaml

resources:

- good-pod-01.yaml

- bad-pod-01.yaml

variables: values.yaml

results:

- policy: ensure-readonly-hostpath

rule: ensure-hostpaths-readonly

resource: bad-pod-01

kind: Pod

result: fail

- policy: ensure-readonly-hostpath

rule: ensure-hostpaths-readonly

resource: good-pod-01

kind: Pod

result: pass

Let’s check the policy with these test cases on Kyverno locally with the Kyverno CLI command.

kyverno apply ensure-readonly-hostpath.yaml -r good-pod-01.yaml -f values.yaml

kyverno test .

Tests for Mutating Policy

This mutating policy will work on a pod resource with a mutate rule to change image tag to digest and a precondition with it. The kyverno apply command will generate a mutate (patchedResource.yaml) file for the resource on which the policy is applied.

Below is the values.yaml , kyverno-test.yaml and patchedResource.yaml file for the mutating policy.

# values.yaml

policies:

- name: resolve-image-to-digest

rules:

- name: resolve-to-digest

values:

resolvedRef: "busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47"

resources:

- name: busybox

values:

request.operation: UPDATE

# kyverno-test.yaml

policies:

- resolve-image-to-digest.yaml

resources:

- pod.yaml

variables: values.yaml

results:

- policy: resolve-image-to-digest

rule: resolve-to-digest

resource: busybox

patchedResource: patchedResource.yaml

kind: Pod

result: pass

# patchedResource.yaml

apiVersion: v1

kind: Pod

metadata:

spec:

containers:

- name: busybox

image: busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47

Let’s check the policy with these test cases on Kyverno locally with the Kyverno CLI command.

kyverno apply resolve-image-to-digest.yaml -r pod.yaml -f values.yaml

kyverno test .

So, this is how to write test cases for Kyverno policies and test them locally.

Oshi Gupta works as a DevOps engineer and technical writer at CloudYuga Technologies. She is a Certified Kubernetes Administrator and was selected for LFX mentorship in Spring 2022 for CNCF Kyverno. She loves writing blogs and is keen to learn...