CI Checks Are Not Enough: Combat Configuration Drift in Kubernetes Resources

In a CI/CD system, CI (Continuous Integration) is the part of the application development process of integrating changes from one or more developers as early as possible, which may happen several times a day, triggering an automated build-and-test sequence for the given project, and ultimately providing valuable feedback to the developer(s) who made the changes. CD (Continuous Delivery) includes infrastructure provisioning and application deployment, which may be manual and consist of multiple stages.

What’s important is that all these processes are fully automated, with each run fully logged and visible to the entire team.

Many security tools for containerized environments are labeled as integrating into CI/CD, but the reality is that CI and CD are different processes that are triggered at different times. DevOps teams need to understand that each is its own separate animal. As such, CD environments need their own specialized solutions to work effectively.

Security in CI

In CI, we would normally employ one or more of the following:

• Security Unit Testing which basically takes a similar approach to unit test but just from a security perspective.
• Source Scan for security best practices, using language-specific linters.
• Static Analysis Security Testing (SAST): A SAST scanner compatible with the programming language used to detect security vulnerabilities in code that you own and in (possibly insecure) libraries that you import.

Containerized applications will also normally employ:

• Container vulnerability scanning which basically scans the application as well as the libraries within the container image for known vulnerabilities. There is a huge difference between vulnerable and exploitable, and container image are notorious for generating many findings which may very well be irrelevant, and oftentimes directly affected by how proficient one is in building slim containers

Kubernetes – Security in CD

DAST (Dynamic Application Security Testing) — a well-established practice in CD —  introspects application in its running state, in many ways just like what an attacker would do. DAST scanners in most cases do not have a language-specific dependency.

With Kubernetes, as the dominating container orchestration platform, the CD can be used to perform a security scan immediately as the final deployment state.

There are so many Kubernetes resources that with all those moving parts you need to have a plan to scan different deployments to detect any security drifts happening in those moving parts. An effective scan of the cluster happens after deployment in order to ensure that assertions still hold in a specific environment from one deployment to another.

This is why CD is not CI, because the context of what you do in CD is different from what you do in the CI phase. In CI you build your application and that’s it. In CD you take those assets and deploy them to a target environment (a test cluster, a staging cluster, a production cluster).

Simply put, in CI you build stuff; in CD you push it out using configuration.

Once deployed, you need to scan the application and two possible negative security changes can occur. First, if you change the configuration it can cause a negative security drift. Secondly, if you change deployment resources, those changes can also cause the same effect. Depending on the culture of the company, changes to configuration parameters can be done by either development or DevOps.

Why CD Matters For Kubernetes Security?

Kubernetes for example has the notion of Mutating Admission Controller that basically can change the deployed resources. For example, it can inject side-car containers just like Istio or Hashicorp Vault implemented — effectively, It can modify pod resources, change labels, etc.

If we, for example, would like to assert whether a Pod has network policy applied to it, an end of CD pipeline after all resources were deployed to the cluster, would be the right point in time to perform such check — simply because network policy associations are based on labels, and if labels are finalized by a Mutating Admission Controller looking at files in Git, or even before deploy may yield an incorrect assertion.

For this very unique Kubernetes (flexible) behavior Network Policy related checks, as well as other resources such as PodSecurityPolicy, Services require analysis at the end of the CD pipeline in order to get an accurate result.

Recipe For Detecting Security Configuration Drifts

Kubernetes as an extremely flexible and powerful orchestration platform has many knobs and flags that control the behavior of deployed workloads and the components that supports those workloads. A practical approach for detecting configuration drift would include the following recipe:

1. For each security flag/knob/configuration
2. Establish an exclusion list of the resources allowed to have this flag/knob/configuration enabled
3. For each deployment, validate that the there are no new exceptions associated of this flag/knob/configuration
4. On Fail
   1. Notify the failed to pass Kubernetes resources
   2. Rollback the deployment if there were critical level hygiene exceptions

Example Guardrails For Your CD

Privileged Workloads

Privileged in Kubernetes wears many faces. Privileged may be one or more of the following:

• Running container(s) as root
• Running containers with additional process capabilities that open system-wide access
• Running Pod that mount the host file system or the container runtime socket
• Running Pod in the host network
• …and more

We would want to apply our security drift detection recipe on each of the above.

Excessive Access Permissions (Kubernetes RBAC)

Excessive RBAC in Kubernetes wears many faces. To name a few:

• Privileges to Create Pods or any resource the eventually creates Pod
• Privileges to read Secrets
• Privileges to create Mutating Admission Controller
• Privileges to create/delete Service resources

Once again, we would want to apply our security drift detection recipe on each of the above

Your Future Self Will Thank You

Don’t make the mistake of thinking CI and CD are one and the same. Save yourself the time, money and headaches of dealing with preventable security breaches by making sure your organization differentiates between CI and CD, and leverages both worlds. CI tools shouldn’t be re-tasked for CD. Perfectly adequate CI tools will either miss things in the CD portion of your pipeline, or flag things that are completely irrelevant. Unless a tool has functionality that is purpose-built for CD, it won’t be enough.

Implementing a CD security solution that has these capabilities and can perform these best practices, whether as a stand-alone solution to augment your existing CI security or as an integration that extends your CI security, will add a critical tool to your existing Secure Development Lifecycle (SDL), particularly for applications delivered using Kubernetes-based platforms.