AWS’s Log4Shell Fix Needs Fixing

Log4Shell, the open source Java logging library Apache Log4j problem has been such a pain in the rump! Now, adding insult to injury, Palo Alto Networks’ security research group Unit 42 discovered several Amazon Web Services (AWS) hot patch fixes came with their own serious security holes.

It’s always something!

So, how bad were these holes? In a word: “Awful.”

Specifically, after installing the patch service to a server or cluster, every container could be exploited to take over its underlying host. For example, if you installed the hot patch to a Kubernetes cluster, every container in your cluster could now escape to higher levels until you either disable the hot patch or upgrade to the fixed version. Aside from containers, unprivileged processes could also be used to exploit the patch to escalate privileges and gain root code execution.

Things Get Worse

Guess what? It gets worse.

Unit 42 reported that containers could escape regardless of whether they were running Java applications, or whether their underlying host runs Bottlerocket, AWS’s hardened Linux distribution for containers. Even if your containers are running with user namespaces or as a non-root user they can still be exploited. Unit 42 assigned Common Vulnerabilities and Exposures (CVE) CVE-2021-3100, CVE-2021-3101, CVE-2022-0070, and CVE-2022-0071 to these vulnerabilities.

Maybe “awful” is too mild a word. Exploited, these are catastrophic holes.

New Fixes

Fortunately, Unit 42 and AWS joined forces to fix the patches with newer, secure patches. So far, no one’s been attacked via these holes in the wild.

If you don’t want to be the first — and if you care about your company or job, this is one “first” you really don’t want — you must install the fixed version ASAP. In fact, now — yes, now — disable would be good.

You do this by installing a permanent fix or the latest hot patch. AWS has already released a fix for each hot patch solution. Once a host runs a fixed version, neither the container escape nor privilege escalation can be used against your system.

Steps to Take

For the hot patches, you need to take the following steps.

1. Version 1.1-16 of the log4j-cve-2021-44228-hotpatch package, which bundles the hot patch service.
2. Version 1.02 of Hotdog, a hot patch solution for Bottlerocket hosts based on Open Container Initiative (OCI) hooks.

There’s not, however, an automated way to use kubernetes-log4j-cve-2021-44228-node-agent Daemonset to install the fixed hot patch. Instead, you’ll need to jump through the following hoops:

1. On standalone hosts, you can upgrade by running

$ yum update log4j-cve-2021-44228-hotpatch

For RPM-based distros or for distros using DEB:

$ Sudo apt install –only-upgrade log4j-cve-2021-44228-hotpatch

1. Update: On Kubernetes clusters that installed the hot patch to every node via the hotpatch Daemonset, you need to manually connect to every node and update the installed hot patch via yum or apt. Note that only deleting the hot patch Daemonset doesn’t remove the hot patch service from your nodes.
2. Hotdog users need to upgrade to the latest version.

Or, if you’re sure — and I mean sure — that your environment is permanently patched against Log4Shell, you can disable Hotdog on a host by running:

$ sudo touch /etc/log4j-cve-2021-44228-hotpatch.kill.,

$ run apiclient set oci-hooks.log4j-hotpatch-enabled=false.

Upgrade Now

Alta Vista’s Prisma Cloud customers can identify affected hosts under the Vulnerabilities tab. The platform detects the hot patch packages and alerts customers on VMs running a vulnerable version.

The reason why this is a top-of-mind problem is that because Log4Shell is such a dangerous bug, users often deployed hot patches at scale. Multitenant container environments and clusters running untrusted images are especially at risk.

Palo Alto Networks, and I, encourage you to upgrade to the fixed hot patch version as soon as possible. But, if you’re still patching against Log4Shell, prioritize that effort first. Yes, this is bad, but Log4Shell can be even worse. So, as I said earlier, get to work patching. It needs doing.

Steven J. Vaughan-Nichols, aka sjvn, has been writing about technology and the business of technology since CP/M-80 was the cutting-edge PC operating system, 300bps was a fast internet connection, WordStar was the state-of-the-art word processor, and we liked it.

Read more from Steven J. Vaughan-Nichols