Linux / Security / Observability / Contributed

An Osquery Field Guide for Log4J Defenders

9 Feb 2022 10:00am, by
Seshu Pasam
Seshu Pasam is chief software architect at Uptycs.

The Log4Shell vulnerability has quickly become one of the most severe vulnerabilities of the last decade. Since public disclosure, defenders have placed countless hours into incident response and remediation. We at Uptycs recently published a blog post on some useful osquery tables to be aware of to help with the investigation, remediation and/or exploit detection cycle for combating the Log4j vulnerability.

In this post, we are sharing a set of queries that work on the open source version of the osquery operating system query tool as well as other helpful validation and remediation techniques.

Log4Shell Brief

The current vulnerability is particularly dangerous since an affected application does not need to be internet-facing; as long as the user-provided input makes its way to the vulnerable application and is processed by the logging library, the application is vulnerable. This means that a well-crafted string sent as part of the HTTP header(s) or HTTP body could allow an attacker to gain remote code execution (RCE). RCE can allow attackers full command and control if they successfully compromise the application.

So which libraries are vulnerable and which version has the fix?

Vulnerable:

  • log4j (1.x) is a legacy library. Even though it has its own vulnerabilities, none are as severe as this.
  • log4j-core jar file from log4j2 library has the vulnerable class.
  • log4j-core-2.0-beta9.jar or later versions are vulnerable.

Most updated version:

  • log4j-core-2.12.4.jar (for JDK7) and log4j-core-2.17.1.jar (for JDK8 and later)

What Inventory and Information Should I Gather?

When it comes to understanding what assets have been potentially affected, we’ll need to quickly gather information and inventory of the environment. We suggest investigating development, staging, test, and production machines as well as Cloud VMs, containers and serverless applications. To further extend the power of osquery, defenders can also use cloudquery and kubequery, two open source tools that Uptycs maintains to fetch relevant data from cloud infrastructure and Kubernetes clusters.

Other assets that should be considered are internal registries/repos, containers registered, blob stores with build artifacts, and employee machines.

What Should I Look for?

Within the aforementioned instances, we suggest looking for information regarding vendor software with Java, vulnerable Java applications, and all artifacts related to log4j version 1 and 2.

Java Processes Running on the Host or in Containers on the Host

Affected JDK/JRE Versions

formatMsgNoLookups=true

Note that changing this does not completely fix the vulnerability. Log4j-core should be upgraded to 2.17.1.

Processes with JVM property -Dlog4j2.formatMsgNoLookups=true

Processes with environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS=true

Get all docker containers with LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable set

Looking for Vulnerable log4j2-core

Processes with vulnerable log4j2-core in command line

Processes with vulnerable log4j2-core opened by Java process

Look for JndiLookup in All Open Jar/War/Ear Files.

Note that this can be an expensive query depending on how many files are open. Also, this can check for jars/uber jars/shaded jars but does not work correctly when checking log4j-core-2.17.1 jar.

More Event Tables

You can use the following event tables to help identify activity related to Java processes post exploit.

--    process_file_events

--    process_events

--    socket_events

How Do I Validate My Fixes?

Validating That JndiLookup.class Was Removed

You’ll want to make sure that this class is no longer loaded by your application.

For each Java process, if jps is available:

Validating Version Upgrade

You may also want to validate that you have properly upgraded to a fixed version. To do so, you can:

  • Get inventory of all jar/war/war/jmod files opened by the process.
  • For each file opened by the process, check if it is or contains vulnerable log4j2-core jar.
  • Validate on an on-going basis to avoid deploying vulnerable applications due to auto scaling, old build artifacts, etc.

There’s plenty more to be done after gathering inventory of your assets. Other security teams should look to implement a WAF to block jndi or ctx strings in user input. EDR/XDR teams should look to monitor processes spawned by Java processes and monitor egress traffic.

Uptycs Threat Research team recently released some research regarding the aftermath of log4j and continues to work feverishly to provide further observations on exploit.

Feature image via Pixabay.