Creating a Trusted Container Supply Chain
When it comes to trust for containers, it’s a two-dimensional problem. First, the full stack of the container, from the base image up, must be verified as secure, with acceptable levels of risk for vulnerabilities and configurations. Second, during each phase of the software development life cycle — from development, through the CI/CD pipeline into deployment — there should be consistent feedback and guardrails for violations of those acceptable levels of risk. When over 99% of exploits use known vulnerabilities, verifying the trust of an image is paramount for closing off a vast majority of exploitable issues.
Let’s look at how this would work if you were a developer building a multiservice eCommerce Kubernetes application at your company.
1st Dimension: Trusted Container Layers
Off-the-Shelf Images
Sometimes you search Docker Hub or other public registries and hit the jackpot. You need to run a machine learning-based recommendation engine for your store, and you find a container image prebuilt with a recommendation algorithm ready to go. The description is perfect and a cursory Google search doesn’t show any complaints. So you pull the image into your Kubernetes cluster and move on.
Not a good idea. When many container images have been found with malware, vulnerabilities or major misconfigurations, this practice could leave your application exposed. If you find that perfect image, running it through a scanner can verify the posture of the image and identify known and unknown malware before you run it in your cluster.
Base Image
For most use cases, an image can get you part of the way there, but then you need to build on top. However, even after building what you need on top, those base images still frequently take up a majority of the container image and running packages. That’s why it’s important to start with a secure base image in the first “From” statement.
For example, let’s say the frontend of your application uses Go, so you copy a frontend you found on GitHub using golang:1.2.0 as the base image. Well, you’ve now already included at least fourteen vulnerabilities, five of which are critical, and you haven’t even added your application components yet.
Instead, if you pulled that base image from a fully vetted registry, you could cut those vulnerabilities out from the start. Imagine if your application is entirely Go-based with 100 microservices. With the off-the-shelf version, you’ve introduced thousands of entry points for a bad actor. With a trusted image, you’ve significantly reduced the attack surface.
Operating System Dependencies and Program Packages
The next step in building container image trust is to include the dependencies. Continuing with the frontend service example, this would be the operating system packages installed with apt, such as ca-certificates to handle certificates and the Go packages installed from the go.mod file, such as the net library to handle networking for your service. These open source libraries, at both an operating system level and language level, also need to be scanned to ensure they do not contain vulnerabilities to prevent exploits. If they do, they need to be updated to later releases, which may have breaking changes that need to be fixed in your code.
Custom Code
If you’ve come this far, by many estimates you’ll have secured over 90% of your service’s code. It’s still important to run your custom code through a code scanner and linter to find the exploits in the code you write, such as SQL injections from not sanitizing your inputs.
Checking each of those layers should happen at multiple stages of your SDLC, to ensure you receive feedback early and often and that things get patched before production.
2nd Dimension: Trust Throughout the SDLC
Development
The most efficient time to remediate vulnerabilities is in development. That’s why it’s important to use a security platform that can provide feedback in dev environments or locally through a CLI. Warning about vulnerabilities and misconfigurations can ensure the image is built to be trustworthy from the ground up.
While you’re building out that frontend service, run it through a scanner to identify the vulnerabilities that need to be fixed. That way, instead of building dependencies on an old release with vulnerabilities, you upgrade first and build on that.
However, the most complete picture of a container is at runtime when it begins interacting with the host and other containers. That’s where image sandboxes come in. Image sandboxing dynamically analyzes running container behavior in an isolated development environment to give a more accurate view of a container image before runtime. The result is knowing about vulnerabilities and misconfigurations so they can be remediated before runtime.
Repository and CI/CD Pipeline
Container images can be vetted before integrating into a repository to ensure they are secure and meet compliance requirements. If you’ve set up an automated testing step in your CI/CD tool, such as Jenkins, code can be blocked with the reason (known vulnerabilities) to prevent vulnerabilities making it into your repository code base.
Let’s say you add a new Go package to your frontend service. You forget to scan it locally due to a deadline. You create a pull request to your Git repository and in minutes you’re told that the new package can’t be committed to the main branch because it contains a critical vulnerability that was patched in a newer version of the package. You’ve been saved from committing vulnerabilities, and you’re given the package version that will work.
Registry
From the repository to the registry, send built images to a trusted and private registry to ensure that images are secure. This is where some companies rely on signing images. However, while a good security practice, it isn’t a perfect solution. Have Kubernetes only pull your frontend container from a trusted registry and block all others. That registry can be continuously monitored for new vulnerabilities to let you know if you need to update your frontend service.
Deployment
Finally, things in the software world move too quickly to remain complacent. In runtime, continuously verify that the containers in use can be trusted using an agent that can identify and alert on new vulnerabilities. If a new vulnerability or misconfiguration is identified that violates your trust policies, fix that image and run it through the trust pipeline again. For those that you can’t patch, use runtime security to block malicious processes and network behavior that attempt to exploit vulnerabilities.
Complete 2D Trust
This two-dimensional approach to verifying container trust dramatically improves the posture of cloud native applications without creating a large amount of additional overhead. Verifying every layer through multiple checks builds confidence that you’ve done the right things to secure your applications.
To learn more about securing containers and other cloud native technologies, consider coming to KubeCon+CloudNativeCon North America 2021 on Oct. 11-15.