One Year of Log4j
You’d think we’d be done with this infamous bug. You’d be wrong.
A year ago, if you’d asked someone what open source Java logging library Apache Log4j was, you’d get a lot of blank glances. Sure, it was used in hundreds, perhaps thousands, of applications ranging from Apache projects, such as Flink, Flume, Hadoop, Kafka, Solr, Spark, and Struts, to Apple iCloud to Elastic LogStash to Twitter, and numerous VMware programs. Heck, it’s even my kids’ favorite Minecraft game. But think about it? Please! It was just some harmless logging program way deep in the code. Oh well, we learned.
The Apache Foundation quietly issued a patch for the Log4j flaw on Dec. 4, 2021. No one paid it much mind. But, then zero-day vulnerability, CVE-2021-44228 — aka Log4Shell, came to light from Mojang Studios’s patch to its Minecraft game on Dec. 10, 2021. It quickly proved way easy to exploit and could easily grab complete control of vulnerable servers. Yuck!
Score of 0.1 to 10
How bad was it? By National Vulnerability Database (NVD) count, it had a 10.0 CVSSv3 score. That’s on a 0.1 to 10 scale. A whole family of attacks, Log4sShell, made this zero-day attack bad enough that it got the White House’s attention. Now that, my friends, is a bug!
According to Check Point, on Dec. 13, 2021, just 72 hours after the initial revelation, there were over 800,000 attack attempts using it. As security company, Nextron Systems’ head of research, Florian Roth, tweeted, “The #Log4Shell vulnerability isn’t just an RCE [Remote Code Execution] 0day. It’s a vulnerability that causes hundreds and thousands of 0 days in all kinds of software products. It’s a 0-day cluster bomb.”
Yes, yes, it was. But a fix was quickly made. By Dec. 20, 2021, the Log4j 2.17.0 library fixed the primary and secondary problems with the code. So, that should have been the end of it. We wish!
It’s Everywhere
It turned out Log4J is freaking everywhere in code. That’s bad. But, what’s far worse is even now, many people don’t have a clue that a vulnerable version of Log4j is still hiding in their code.
By January 2022, Microsoft warned of continued attempts by nation-state actors and cybercriminals to exploit the vulnerabilities found in Log4j to deploy malware on vulnerable systems. At the same time, Check Point researchers discovered an Iran-linked threat actor, APT35, exploiting the Log4j vulnerability in order to deploy modular, PowerShell-based malware. This same tactic is being used to this very day. Microsoft’s research team also spotted a China-based group that exploited the vulnerability in some versions of the VMware Horizon. Here the goal was to install Night Sky ransomware.
And, of course, the flaw was also being used by ordinary crooks planting crypto-mining malware. What’s a security hole for these days if not to try to rip off computing power to literally make money?
Earlier in December 2022, Recently, the Cybersecurity and Infrastructure Security Agency (CISA) revealed not only were hackers still using Log4Shell, but Iranian government agents had used it to pry open Federal government agency VMware systems.
Still, that must be an outlier, right? We’re finally ahead of the problem, aren’t we? Aren’t we!?
Nope.
72% of Organizations Remain Vulnerable
According to Tenable, a security company, “72% of organizations remain vulnerable to the Log4Shell vulnerability as of Oct. 1, 2022.” Why!? “Nearly one-third (29%) of these assets had recurrences of Log4Shell after full remediation was achieved.”
So, yes, the code is fixed, and then someone introduces “new” code that pulls in an old broken version of Log4J. And, presto! Instance vulnerability!
As Bob Huber, Tenable chief security officer, said, “Full remediation is very difficult to achieve for a vulnerability that is so pervasive, and it’s important to keep in mind that vulnerability remediation is not a ‘one and done’ process. While an organization may have been fully remediated at some point, as they’ve added new assets to their environments, they are likely to encounter Log4Shell again and again. Eradicating Log4Shell is an ongoing battle that calls for organizations to continually assess their environments for the flaw, as well as other known vulnerabilities.”
Dependencies, Dependencies, Dependencies
How is this possible? Tenable doesn’t go into it, but Endor Labs’ Station 9 study, The State of Dependency Management, gives us a huge hint. In many vendor open source codebases, 95% of the vulnerabilities are not selected by developers but indirectly pulled into projects.
“By some measures, for every one dependency a developer brings into a software project, there are, on average, 77 to 78 transitive dependencies,” said Varun Badhwar, Endor Labs’s co-founder and CEO. So, “95% of vulnerabilities found are in those transitive dependencies, the things that came with the things you brought. We need to track all of this in our environment and understand which apps these packages are being used in.”
The need for Software Bills of Material (SBOM) and Supply chain Levels for Software Artifacts (SLSA) has never been clearer. Obviously, companies still don’t know what’s in their code before they deploy it.
Even the fact that, by Tenable’s count, 28% of organizations across the globe have fully remediated Log4Shell as of Oct. 1, 2022, a 14-point improvement from May 2022, isn’t that reassuring.
Endemic Vulnerability
What that means Bob Burns, Thales Chief Product Security Officer, said Log4j is proving to be an “endemic vulnerability as it will remain in systems for years to come.” It also has caused people to worry about the essential security of open-source software. Of course, proprietary software, as the SolarWinds fiasco showed, has its own problems.
Regarding proprietary software security woes, Charlie Jones, a security company ReversingLabs Software Assurance Evangelist, expects the impact of Log4Shell to rival that of MS-17-10. This was the “Eternal Blue” Microsoft SMB flaw (MS-17-10) that powered the NotPetya and WannaCry wiper malware attacks. But “Log4Shell presents an even more complex challenge than MS-17-10, as it tends to be deeply nested within application dependencies and therefore more difficult to quickly identify with standard tooling.”
The real lesson of Log4j, even as we struggle to put a final end to it, is the urgent need to know exactly what we’re putting into our programs. Until we have this fundamental software supply chain resolved with automated tools, we can expect to see more Log4j problems pop up. Let’s just hope that none of them are as bad as Log4j has proven to be!
