What news from AWS re:Invent last week will have the most impact on you?
Amazon Q, an AI chatbot for explaining how AWS works.
Super-fast S3 Express storage.
New Graviton 4 processor instances.
Emily Freeman leaving AWS.
I don't use AWS, so none of this will affect me.
Containers / Kubernetes / Open Source / Security / Software Development

Container Camp: Four Security Lessons from a Live Container Hacking Session

Oct 9th, 2017 2:00am by
Featued image for: Container Camp: Four Security Lessons from a Live Container Hacking Session

The Demo Gods were merciful at this year’s Container Camp in London. When doing a live hacking battle on stage, the odds of crashing are extremely high. But for 40 minutes the Demo Gods shined on Andrew Martin as he played Capture the Bitcoin and hacked right through Ben Hall’s containers.

It all started with Hall, founder of the online learning service Katacoda, making this call to arms: “Andrew, can you provide a security overview of my system? If you manage to break in, you could gain a bitcoin.”

Sadly, we can’t offer you a bitcoin — worth almost $4,000 at the time of printing — for cracking out of a container, but we can show you the very common container configuration mistakes Hall purposely made and how Martin, who is starting a security engineering company and is currently a DevOps lead for the U.K. Home Office, took advantage of them.

One thing to mention before we dive right in, the bitcoin was divvied up into five pieces, to lure Martin to find them in different parts of Hall’s system in the short, no-pressure-at-all, on-stage and live experience. Hence five ways to keep from getting hacked. Enjoy!

How a Hacker Looks at Your Containers

Talking about the experience, Hall explained, “I didn’t really know how Andy would attack, but I knew what he was going to be looking for and his aim.”

Martin told the crowd that he started by looking at the Linux boxes and each box could be anywhere, and from there try to exploit the system by any means necessary. Three main ways he can try to do that are:

  1. Enumerating the external interface — in this case the internet — and penetrating that security boundary with a remote code execution (RCE) — the application running code and doing something it shouldn’t do.
  2. Identify what privilege is given to run our code on that server — and if it’s restricted, exchanging to a privileged user.
  3. And, in the case of containers, breaking out.

“A system is supposed to give as little information to an attacker as possible,” Martin said, but that isn’t always the case. So here are the kernels he was hunting…

Container Hack #1: Privileged Containers are Simply Not Contained

Martin started out by leveraging a vulnerability called SambaCry that hacking group Shadow Brokers exfiltrated from the U.S. National Security Agency. The first exploit was similar to the bug that occurred in the WannaCry attack that got the NHS (U.K. National Health Service) and telecom giant Telefonica. He didn’t run the same code, but a similar protocol attack.

“We had that running in a Linux container ostensibly so the Linux machine could talk to the Windows machine. Because that was vulnerable and I was able to probe it for Samba shares,” Martin said.

He scanned for Samba shares, found one, ran a RCE exploit, and entered the container.

“I still had to find a way to become the root user because that’s who had the file system permissions for the bitcoin wallet,” he said.

Martin started by using the amicontained container introspection tool.

“You can run it on any machine and see how secure you are,” Hall explained, calling it “a really nice quick win and really convenient for Andy to see how much access he has.”

It turned out to be a privileged container. The –privileged flag gives all capabilities to the container, and it also lifts all the limitations enforced by the device cgroup controller. Essentially, the container can then do most everything that the host can do.

According to StackOverflow, “This flag exists to allow special use cases, like running Docker within Docker,” but that was not the use case here.

Martin uncovered via amiccontained that “It’s a privileged container and running this tool could show all the kernel capabilities, which then suggests checking what devices are available in /dev.”

From here, he discovered a very important kernel cap_sys_admin call “and that’s carte blanche to do almost anything,” Martin said.

“It was really important to highlight that when people can’t get things to work inside containers, they just stick privileged on the containers and leave it,” Hall said.

This is a common mistake developers make where they take down the security by making the container privileged because it’s the fastest way to make the container behave like you hope it would. This was something done in the early days of older containers, but you never want to run privileged containers, particularly not on an Internet-exposed service.

“At best privilege mode is to prove the concept of things working, but it’s so dangerous and there are so many caveats,” Hall warned.

“I’d say at best it is good to prove that something is possible to do inside a container because you can disable all the security individually. Actually, the best is to drop all permissions and then give back what you want.”

At the point that Martin had accessed it, the container had been started with the –privilege flag, which means a huge amount of security is disabled and most namespaces are shared.

Apply –no-new-privileges to make sure users can’t get new privileges inside the container, and avoid –privilege.

Container Hack #2: Don’t Leave Apps Lying around Your Container

Next, Martin tried to mount the host’s file system into the container.

“The secret exists outside of the container, so because we’re in the container we want to get outside to the host. In order to do that, we can mount a partition, but the standard Linux security model still applies in a container — I can only do that if I’m the root user inside the container, but I was ‘Nobody’,” which meant Martin was denied the ability to mount.

At that point, Martin, in his role as hacker, would have to become the root user inside the container in order to mount the host file system in the container.

He explained, “In this attack, we aren’t looking to escape the container but instead trying to pull information on the host into the container to read or edit it, basically get complete access.”

In a normal container-less situation, by becoming the root user, he could pull in customer data including passwords and gain access to servers.

“At this point, we’re searching for a way to become a root user inside a privileged container, then we have equivalent permissions to root on the host machine.

“The fundamental thing is that once you have rights access to the file system, then you can add your credentials to any user and access the host directly.” Martin said, “And then it’s game over.”

So Martin used a privilege checking script — linuxprivchecker — to check that the whole system is configured correctly and if the right file permissions are in place.

“Because there are multiple admins on a computer, if one person goes in to fix something and then another person does something else, there’s no coherent state unless they’re communicating every keystroke. This means misconfigurations become more likely, and is a good reason to deploy immutable infrastructure,” Martin said.

He discovered that Hall had a debugging application still running inside the container that was ripe for exploitation.

Hall explained his purposeful mistake: “People always leave stuff in containers because it makes life easier. In this example, I had additional tooling inside the container which wasn’t there for the application but just for me, so it could be repurposed. If you are thinking in a container security-minded way, you would remove anything excess.”

The debug application was then used to get root access inside the container.

“And once we’ve got root inside the container, it’s possible to mount the host filesystem in and read the contents of the file which contains the bitcoin.”

And another piece of bitcoin gained.

Next up Martin went for an easier exploit by using nmap to scan for the presence of a Shellshock — sometimes also called Bashdoor — vulnerability, a family of security bugs that became popular a couple years ago.  Nmap detects this vulnerability by executing a command that prints a random string and then attempts to find it inside the response body.

Martin was able to remotely exploit this vulnerability to run code in the container.

He said, “It’s not possible to defend network or protocol attacks with containers because the vulnerability is at the application level — it’s like sticking your head above the parapets, people can still attack what they can see and get in over the walls of the container/castle. Container defense comes in stopping them from penetrating any further into the castle. Shellshock is a remote code execution — we used it to get in the container and start running the commands.”

Hall added, “Basically, when we send a request to the server, you can send additional data in HTTP headers — such as the browser you’re running. In the case of Shellshock, this allows any command to run on the system. Andy used this to find a particular socket file belonging to Docker and access the Docker engine daemon. Once you have access to the Docker daemon, then you have access to all the systems.”

Then, once a perpetrator can write his own commands, he can run his own container with volume mounts from the host and access anything.

And a second piece of bitcoin achieved.

Container Hack #3: Build Layers into Your App to Build up Your Defense

There’s no doubt PHP is one of the most popular programming languages. Part of this is that it’s a simpler language to learn that more people know, which means building anything with PHP usually cuts on your staffing costs. But it also tends to be more vulnerable.

To offer a commonplace example, Hall hosted a sample application running a form with a particular vulnerability which it could execute arbitrary code in the background.

He explained that “At a high level, it used a PHPMailer functionality with a known issue where you could send it a particular request which would allow you access to the system.”

So Martin could take advantage of this to “create a backdoor via the exploit. We start through the front door — the web application — something runs on the server through the vulnerable PHPMailer and that places another file on the disk. By making HTTP requests to that new file we can run code.”

Hall says the best way to fight back at the inherent security problems is by building more complex security layers around your containers.

“For defensibility, it’s all about adding layers, and from an attack side, it is about reducing layers,” Hall said.

In a similar attack, he said “the hackers found what the PHPMailer looked like which gave them access to one part of the system, which gives them a starting point to access to rest. And then at that point: Game over. All these weird looking strings get you access to important parts.”

Martin eagerly added: “The most interesting part of hacking is running a command that looks absolutely ridiculous which is, in fact, a magical incantation that opens a secret door on the target system”

In this live-hacking case the PHP bug was running on a migrated Kubernetes with the default pre-RBAC security settings, “nothing turned off, nothing really locked down.”

Hall explained that “Because of that, with Kubernetes 1.5, you get a special token which allows you access to the Kubernetes cluster which again gives you access to everything.”

This is something Kubernetes evidently did to look and feel more like it’s Google grandparent, Borg.

As Martin said, “The difference is that internally at Google these service accounts are locked down, but Kubernetes is an open-source project and security is often takes second place to features. This is the right thing to do for product development and user acquisition, and everybody does it, but, practically, its makes security more difficult.”

“Every project has that danger of prioritizing security second,” Hall echoed.

But while speed of feature delivery is typically prioritized over security, you can usually supplement your open-source work with paid-for service. Hall offered Aqua as a security tool that raised millions simply because containers are hacked every day.

“If you can’t depend on the open source project you have to depend on vendors to add the extra model,” Hall said.

Nevertheless, Aqua was not used in this situation, the PHPMailer was infiltrated, and the third piece of Bitcoin found.

Container Hack #4: Keep it Updated and Use an Intrusion Detection System

Next, Martin used the Kubernetes container orchestration engine to run another attack: Dirty COW, a privilege escalation vulnerability in the Linux kernel.

Hall called it “a particular problem because it attacked a low-level part in a Linux kernel, while most hackers go up and out of the container, this one went down and deep into the kernel and allowed it to execute arbitrary code. You could have done all the right settings, had everything optimized, but if you were running this version of the kernel, you are still exposed, highlighting the dangers of not patching.

“Even if you think you’re in a highly secure configuration, you’re still vulnerable if you don’t patch the host machine,” he said.

You always have to remember to update the security strength of the systems used.

But in this situation — and in the interest of time on stage and also not losing loads of money — Hall blocked Martin by updating his kernel security settings. Of course, he knew he was being attacked and how just then.

There are intrusion detection systems or IDS that can help detect and alert you to these security breaches, which the pair recommend. For containers, you need a container-native IDS, like:

  • Aqua and Twistlock are proprietary, closed-source software that can detect and shut down attacks in progress
  • Sysdig is more observability and monitoring and an open-source intrusion detection tool — laser pinpoints each container that has a violation

“You can tune them to be a lot more sensitive so you can detect all sorts of anomalous behavior very quickly. While typically IDS would rely on honeypots and brute force login detection, with containers and namespaces, you can see if someone tried to run a shell inside a container or is running a program that’s not whitelisted,” Martin explained.

If You’re One of the Good Guys, Why Hack at All?

Both Hall and Martin are enthusiastic container users and container safety buffs.  As Martin concluded: “Defending is so much more difficult than attacking because it’s an asymmetrical battlefield, and you have to have a watertight defense. The things we demonstrated were generally problems with the Linux configuration.”

Containers bring with them their own slew of problems.

“Before containers, you would have had to configure the whole host for anything running on it. Now you can customize the perimeter of each container with a different security configuration.” Martin added that “Kubernetes security is improving with every release but [it] is still a moving target.”

“This is one of the motivations for Katacoda,” Hall said, referring to his online learning service, which has content dedicated to Docker and container security. “Everything is moving at such a fast pace, it’s critical that developers have a place where they can go to learn and experiment with these new technologies, in particular, the security features in a safe environment. If it all goes wrong, they can hit refresh and be given a brand new environment.”

Photo: Paul Bouwer

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Docker, Simply, Sysdig.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.