An Osquery Field Guide for Log4J Defenders

The Uptycs guide to using OSQuery to ferret out Log4J vulnerabilities.

Feb 9th, 2022 10:00am by Seshu Pasam

Featued image for: An Osquery Field Guide for Log4J Defenders

Feature image via Pixabay.

Seshu Pasam is chief software architect at Uptycs.

The Log4Shell vulnerability has quickly become one of the most severe vulnerabilities of the last decade. Since public disclosure, defenders have placed countless hours into incident response and remediation. We at Uptycs recently published a blog post on some useful osquery tables to be aware of to help with the investigation, remediation and/or exploit detection cycle for combating the Log4j vulnerability.

In this post, we are sharing a set of queries that work on the open source version of the osquery operating system query tool as well as other helpful validation and remediation techniques.

Log4Shell Brief

The current vulnerability is particularly dangerous since an affected application does not need to be internet-facing; as long as the user-provided input makes its way to the vulnerable application and is processed by the logging library, the application is vulnerable. This means that a well-crafted string sent as part of the HTTP header(s) or HTTP body could allow an attacker to gain remote code execution (RCE). RCE can allow attackers full command and control if they successfully compromise the application.

So which libraries are vulnerable and which version has the fix?

Vulnerable:

log4j (1.x) is a legacy library. Even though it has its own vulnerabilities, none are as severe as this.
log4j-core jar file from log4j2 library has the vulnerable class.
log4j-core-2.0-beta9.jar or later versions are vulnerable.

Most updated version:

log4j-core-2.12.4.jar (for JDK7) and log4j-core-2.17.1.jar (for JDK8 and later)

What Inventory and Information Should I Gather?

When it comes to understanding what assets have been potentially affected, we’ll need to quickly gather information and inventory of the environment. We suggest investigating development, staging, test, and production machines as well as Cloud VMs, containers and serverless applications. To further extend the power of osquery, defenders can also use cloudquery and kubequery, two open source tools that Uptycs maintains to fetch relevant data from cloud infrastructure and Kubernetes clusters.

Other assets that should be considered are internal registries/repos, containers registered, blob stores with build artifacts, and employee machines.

What Should I Look for?

Within the aforementioned instances, we suggest looking for information regarding vendor software with Java, vulnerable Java applications, and all artifacts related to log4j version 1 and 2.

Java Processes Running on the Host or in Containers on the Host

SELECT *
FROM processes
WHERE name LIKE 'java%'

SELECT *

FROM processes

WHERE name LIKE 'java%'

Affected JDK/JRE Versions

SELECT *
FROM deb_packages
WHERE name LIKE '%jdk%'
      OR name LIKE '%jre%';


SELECT *
FROM rpm_packages
WHERE name LIKE '%jdk%'
      OR name LIKE '%jre%';

SELECT *

FROM deb_packages

WHERE name LIKE '%jdk%'

OR name LIKE '%jre%';

SELECT *

FROM rpm_packages

WHERE name LIKE '%jdk%'

OR name LIKE '%jre%';

formatMsgNoLookups=true

Note that changing this does not completely fix the vulnerability. Log4j-core should be upgraded to 2.17.1.

Processes with JVM property -Dlog4j2.formatMsgNoLookups=true

SELECT *
FROM processes
WHERE cmdline LIKE '%-Dlog4j2.formatMsgNoLookups=%'

SELECT *

FROM processes

WHERE cmdline LIKE '%-Dlog4j2.formatMsgNoLookups=%'

Processes with environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS=true

SELECT *
FROM process_envs
WHERE key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS'

SELECT *

FROM process_envs

WHERE key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS'

Get all docker containers with LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable set

SELECT *
FROM docker_containers
WHERE env_variables like '%LOG4J_FORMAT_MSG_NO_LOOKUPS%'

SELECT *

FROM docker_containers

WHERE env_variables like '%LOG4J_FORMAT_MSG_NO_LOOKUPS%'

Looking for Vulnerable log4j2-core

Processes with vulnerable log4j2-core in command line

SELECT *
FROM processes
WHERE cmdline LIKE '%log4j-core%'
      AND cmdline NOT LIKE '%log4j-core-2.17.1%'

SELECT *

FROM processes

WHERE cmdline LIKE '%log4j-core%'

AND cmdline NOT LIKE '%log4j-core-2.17.1%'

Processes with vulnerable log4j2-core opened by Java process

SELECT *
FROM process_open_files o
JOIN processes p
USING (pid)
WHERE <a href="http://p.name/"><span class="s1">p.name</span></a> LIKE 'java%'
      AND o.path LIKE '%log4j-core%'
      AND o.path NOT LIKE '%log4j-core-2.17.1%'

SELECT *

FROM process_open_files o

JOIN processes p

USING (pid)

WHERE <a href="http://p.name/"><span class="s1">p.name</span></a> LIKE 'java%'

AND o.path LIKE '%log4j-core%'

AND o.path NOT LIKE '%log4j-core-2.17.1%'

Look for JndiLookup in All Open Jar/War/Ear Files.

Note that this can be an expensive query depending on how many files are open. Also, this can check for jars/uber jars/shaded jars but does not work correctly when checking log4j-core-2.17.1 jar.

SELECT *
FROM yara
WHERE count &gt; 0
  AND sigrule = 'rule class { strings: $cls = "JndiLookup" condition: $cls }'
  AND path IN (SELECT path
              FROM process_open_files
              WHERE path LIKE '%._ar')
  AND path NOT LIKE '%log4j-core-2.17.1%'

SELECT *

FROM yara

WHERE count > 0

AND sigrule = 'rule class { strings: $cls = "JndiLookup" condition: $cls }'

AND path IN (SELECT path

FROM process_open_files

WHERE path LIKE '%._ar')

AND path NOT LIKE '%log4j-core-2.17.1%'

More Event Tables

You can use the following event tables to help identify activity related to Java processes post exploit.

-- process_file_events

-- process_events

-- socket_events

How Do I Validate My Fixes?

Validating That JndiLookup.class Was Removed

You’ll want to make sure that this class is no longer loaded by your application.

For each Java process, if jps is available:

for i in `jps | grep -v Jps | awk ‘{print $1}’`; do jcmd $i GC.class_histogram | grep -q JndiLookup &amp;&amp; { echo “PID $i MIGHT be vulnerable”; }; done

1	for i in `jps \| grep -v Jps \| awk ‘{print $1}’`; do jcmd $i GC.class_histogram \| grep -q JndiLookup && { echo “PID $i MIGHT be vulnerable”; }; done

Validating Version Upgrade

You may also want to validate that you have properly upgraded to a fixed version. To do so, you can:

Get inventory of all jar/war/war/jmod files opened by the process.
For each file opened by the process, check if it is or contains vulnerable log4j2-core jar.
Validate on an on-going basis to avoid deploying vulnerable applications due to auto scaling, old build artifacts, etc.

There’s plenty more to be done after gathering inventory of your assets. Other security teams should look to implement a WAF to block jndi or ctx strings in user input. EDR/XDR teams should look to monitor processes spawned by Java processes and monitor egress traffic.

Uptycs Threat Research team recently released some research regarding the aftermath of log4j and continues to work feverishly to provide further observations on exploit.