Azure DevOps Featured

Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability

Azure DevOps can be configured with advanced Code Search. That feature relies on Elastic Search. Depending on the age of your server, JVM version and Elastic Search version this may result in your setup being vulnerable to CVE-2021-44228.

Jesse Houwing

Dec 13, 2021 • 6 min read

Update: Official patches are now available:

Update: Official statement from Microsoft:

Azure DevOps Server ships with a pretty old version of Elastic Search and by default installs a non-auto updating JVM. I just checked the version of Elastic Search that ships with my version of Azure DevOps Server and it's 6.2.4. That version ships with log4j 2.9.1, which is unfortunately too old for the simple fix described by Elastic to do the trick:

Elasticsearch 6 and 7 are not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager. Elasticsearch running on JDK8 or below is susceptible to an information leak via DNS which is fixable by the JVM option identified below. This option is effective for Elasticsearch versions 5.6.11+, 6.4+, and 7.0+.

Elasticsearch 5 is susceptible to both remote code execution and an information leak via DNS. For versions 5.6.11 - 5.6.16, this can be mitigated by setting the JVM option. Users on an earlier version of 5.x, are recommended to upgrade to 5.6.16. We are exploring other options for cases where upgrading is not possible. Please note that while we provide these remediations, Elasticsearch 5 is not a supported version, and we always recommend updating to the latest release.

https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

For TFS 2017 RTM, Elastic Search 1.7.1, which ships with log4j 1.2.17, which is not vulnerable.
For TFS 2017 Update 1 - TFS 2018 Update 1, Elastic Search 2.4.1, which ships with log4j 1.2.17, which is not vulnerable.
For TFS 2018 Update 2+, Elastic Search 5.4.1, which ships with log4j 2.8.2, which is vulnerable and can not be resolved by setting the JVM option.
For Azure DevOps Server 2019, Elastic Search 6.2.4, which ships with log4j 2.9.1, which is vulnerable and can not be resolved by setting the JVM option.
For Azure DevOps Server 2020, Elastic Search 6.2.4, which ships with log4j 2.9.1, which is vulnerable and can not be resolved by setting the JVM option.

Luckily the default installation mode only binds Elastic Search to the local loopback network interface limiting access from the outside world, but if you've setup Search on a separate VM and didn't follow the Securing Search guidance to limit access you may want to look into that right now.

You currently have 3 options to deal with this issue:

Remove Search from your installation
Patch the log4j-core-*.jar files on disk
Patch the log4j vulnerability at runtime

I also highly recommend you update the JVM that was installed when you configured Search. The default option does not auto-update and chances are the one you're running is already quite old.

Remove Search

The simplest most secure solution at the moment is to uninstall Azure DevOps Search from your Application Tiers and disabling/uninstalling the Elastic Search service:

Or remove the service altogether:

And remove the Search feature from the Azure DevOps Administration Console:

Remove Azure DevOps Search Service from the Azure DevOps Server Admin Console

Patch Elastic Search files Instead

If you don't want to turn Search off, you'll need to patch log4j-core to strip the vulnerable classes from the jar files. You can remove the vulnerable classes from the jar file with a single command. All of the docs I found give you this command which will work on Linux:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

On Windows that command won't work, since the zip command isn't available. If you've got 7-zip installed, use the following the command for Azure DevOps Server 2020:

cd "C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start

To resolve the issue for Azure DevOps Server 2019+:

cd "C:\Program Files\Azure DevOps Server 2019\Search\ES\elasticsearchv6.2\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start

To resolve the issue for Team Foundation Server 2018 Update 2+:

# cd "C:\Program Files\Microsoft Team Foundation Server 15.0\Search\ES\elasticsearch-5.4.1\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start

Alternatively, patch log4j at runtime

You can also install log4j-jndi-be-gone and instruct Elastic Search to load it to remove the vulnerability at runtime. This will also work for older versions of Team Foundation Server and Azure DevOps Server.

Download the log4j-jndi-be-gone-1.0.0-standalone.jar from their GitHub repository and save it to your server.

Save the jar file where Elastic Search can find it

Run elasticsearce-service.bat manager to open the configuration panel for Elastic Search and add the jar file to the configuration:

-javaagent:C:\AzureDevOpsData\log4j-jndi-be-gone\log4j-jndi-be-gone-1.0.0-standalone.jar

Add log4j-jndi-be-gone agent to the Elastic Search configuration

Then restart the Elastic Search service:

Restart Elastic Search after adding log4j-jndi-be-gone.

Update the Java Runtime Environment for Search

While you're at it, update the JVM to the most recent version. You can find the latest Java 8 Runtime here.

Note
If you choose to use Azul Zulu OpenJDK, ensure that you download version 8.

Extract the zip file into the AzureDevOpsData\Java\ folder on your Application Tier: