Log4J – A 10 step mitigation plan

Originally posted on the Xebia Blog.

Update: which vulnerabilities are present?

The initial blog below was written around CVE-2021-45046 and CVE-2021-44228, in which Log4J its JNDI functionalities could be misused in some cases to get to an RCE (Remote Code execution). The recommendation back then was to upgrade to Version 2.12.2 when you are on Java7 and version 2.16.0 in all other cases. Now a new vulnerability is discovered: CVE-2021-45105 . This allows for a DoS (Denial of Service) attack on Log4J when you use the log4j-core JAR in combination with a specific configuration which involves using Context Lookups like ${ctx:loginId}or $${ctx:loginId} according to the post by the Apache Log4J team. There are 2 fixes for this:

• Update the Log4J core JAR to version 2.17.0;
• Update your Log4J configuration as described here by the Apache Log4J team.

We recommend to FIRST make sure you have no systems running on a version of Log4J core below version 2.16.0 (or 2.12.2 in case of Java7). If you do, patch them to the latest available version (2.17.0 for Java 8 and above, 2.12.2 for Java 7). Lastly, given that the recent security news attracted a lot of security researchers their attention to various open-source libraries, we expect that more CVEs on various libraries will be reported soon.

What to do first?

1. Check if you are affected by the vulnerability

There are various ways in which you can check whether you are vulnerable: By means of static analysis:

• If you write software that runs on the Java Virtual Machine (JVM) based on Java/Scala/Kottlin/Groovy/Clojure check whether you are using Log4J-core JAR with a version below 2.16.0.
• Check what other software you are running and see if the software is vulnerable according to the list published by the NCSC-NL. CISA.gov has a similar registry.

By means of dynamic analysis:

• There are various tools, nicely again enumerated by NCSC-NL at their Github repository with which you can detect whether you are vulnerable. See which one you like best given the situation you are in.

2. Patch! Patch! Patch!

Once you found vulnerable software, patch it! When you write your own software, it means: update Log4J-core at least to version 2.17.0 (2.12.2 when using Java7).When you work with third party software that contains a vulnerable version of Log4J, then upgrade the given software or follow the patch procedure outlined by your vendor. If there is no upgrade or guidance available, contact the vendor of the software and ask for their support. Does that not help either? Read on to check on other possible solutions.

3. What if you cannot patch?

If you cannot upgrade the library, there might be various solutions to the problem:

• To prevent the Remote Execution attack, remove the class from the application which actually causes the RCE problem. Or better phrased: introduces the vulnerability. In this case, that is the JndiLookup.class file. This file needs to be deleted from the JAR file. You can delete the file easily by means of tools like Zip and 7-Zip as the JAR file is basically a “zip file” with java .class files and additional resources. One way to do this is by using 7-zip: “c:\Program Files\7-Zip\7z.exe” d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class . The Log4J team advises to execute the following command when using a Mac or Linux:  zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class .
• Another approach involves applying configuration changes to the Log4J library to make the vulnerable library safe again. However, these required changes differ per version of the library in use. These configuration changes might actually not always be effective and sometimes come with unwanted side-effects. You can best read up about it here.
• Take the vulnerable system offline and try to find a workaround for the functionality provided by the system until there is a patch.
• There are runtime patches such as log4j-jndi-be-gone that can be injected into a process, which stop loading of the JNDI relevant classes. This doesn’t require any changes to application files in case they’re signed, or otherwise not editable, plus these tools will also remove the vulnerability in case the vulnerability is picked up from another directory due to misconfiguration (class path).

But what about the WAFs?

We sometimes hear that it is recommended to use a Web-application-firewall (WAF) to prevent attackers to exploit the vulnerability. Though a WAF can help as a defense-in-depth control, it should not be your primary and only defense. A WAF helps to delay an attacker: when configured correctly, it can stop many of the exploits that are currently found in the wild. However, a sophisticated attacker might still be able to bypass the WAF. Therefore a WAF can help as a temporal stopgap while you have vulnerable components, but we recommend to use this as a second layer of defense. The actions described at 2 & 3 should be your primary defense.

A little note: when you write your own software and use JAR-shading

Jar-shading is a technique which is sometimes applied by library/tool creators. This process involves packaging multiple – often supportive- JAR files together, which are then renamed in order to ensure that the authors of the library/tool have better control over the dependencies used. This means that the Log4J JAR that is vulnerable might be present in the library/tool without being easily detected as its name is changed. In that case, solutions like the earlier mentioned log4j-jndi-be-gone might help as well.

What to do next?

There are a few things you need to take care of:

4. Security and Event Monitoring

Make sure you have a proper logging and Security Event Monitoring (SEM) system in place. Make sure you are able to detect when the Log4Shell exploit is applied at your system and ensure you can detect odd behavior at your system in scope. Want to get started with SEM? There are plenty of mature solutions in your cloud provider and mature open source tools available for your stack.

5. Prepare for Incident Response

Always be ready to do incident response. Make sure you have a run-book and a call–tree ready when security incidents happen. If you think/detect that your application landscape is successfully exploited, contact a professional incident response team.

What about the future?

There are a few things you can do to speed up your response to similar future events:

6. Always know what you are running: the Software Bill of Materials

Make sure you know what you are running on your platform. The Software Bill of Materials (SBoM) describes all the various software components on which your system is based. If you keep an active track of your SBoM with tools like OWASP dependencyTrack, it becomes easier to know whether software you are using is vulnerable. Additionally there are great open-source tools, like the OWASP Dependency Checker, Trivy, Clair, and many others which you can use as part of your CI/CD pipeline to detect whether some of the software you are building has known vulnerabilities.

7. Stay up to date automatically

When you write or compose your own software: use tools like DependaBot, Renovate, Snyk, etc to automatically create pull/merge-requests and compatibility scores. It’s much easier to apply a patch to a system when it uses a recent version of a library. Some of these tools provide a compatibility score and guidance on how to solve breaking changes.

8. Understand the risks of your own application landscape: Document and Threat-model

When vulnerabilities rise at your system, it is good to understand what risks are involved when these get exploited. For this, it can help to document the overall architecture of your application landscape and do active threat-modelling.

9. Isolate your applications and services

Software will always contain vulnerabilities. This is why you need to isolate your software to make sure that an exploit of one vulnerable component does not automatically mean that the full system is compromised. Isolation can be done in various ways:

• Implement network segregation and separation. Make sure that egress and ingress filtering is in place and that network traffic is logged in order to be analyzed by SEM software.
• Limit components their runtime capabilities: make sure that they do not have to execute privileged actions, and ensure that proper containerization takes place.

10. Remain blameless: collaborate and evolve

Building software, maintaining software and operating the complex systems that we use today, are no easy task. This means that mistakes will be made and unforeseen things will happen. These events can often lead to security incidents if not detected early. In order to respond to these events quickly, it is key to collaborate in a blameless culture in order to ensure steady communication on any of the mistakes made. It matters how you react to unforeseen events, and improve on what you have learned. That is how you improve your resilience as a person, a team and as an organization.

Update 18 December :

Added section “Update: which vulnerabilities are present?”. Updated section 2, 3, and 7: made the wording a bit more clear,  added the various CVEs to make a clear distinction between the problems at hand, and provided the focus on upgrading to version v2.16.0 (or v2.12.2 for Java 7) and lesser on the DoS issue fixed in v2.17.0/v12.12.3.