Update: Official patches are now available:
Update: Official statement from Microsoft:
Azure DevOps Server ships with a pretty old version of Elastic Search and by default installs a non-auto updating JVM. I just checked the version of Elastic Search that ships with my version of Azure DevOps Server and it's 6.2.4. That version ships with log4j 2.9.1, which is unfortunately too old for the simple fix described by Elastic to do the trick:
Elasticsearch 6 and 7 are not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager. Elasticsearch running on JDK8 or below is susceptible to an information leak via DNS which is fixable by the JVM option identified below. This option is effective for Elasticsearch versions 5.6.11+, 6.4+, and 7.0+.
Elasticsearch 5 is susceptible to both remote code execution and an information leak via DNS. For versions 5.6.11 - 5.6.16, this can be mitigated by setting the JVM option. Users on an earlier version of 5.x, are recommended to upgrade to 5.6.16. We are exploring other options for cases where upgrading is not possible. Please note that while we provide these remediations, Elasticsearch 5 is not a supported version, and we always recommend updating to the latest release.
- For TFS 2017 RTM, Elastic Search 1.7.1, which ships with log4j 1.2.17, which is not vulnerable.
- For TFS 2017 Update 1 - TFS 2018 Update 1, Elastic Search 2.4.1, which ships with log4j 1.2.17, which is not vulnerable.
- For TFS 2018 Update 2+, Elastic Search 5.4.1, which ships with log4j 2.8.2, which is vulnerable and can not be resolved by setting the JVM option.
- For Azure DevOps Server 2019, Elastic Search 6.2.4, which ships with log4j 2.9.1, which is vulnerable and can not be resolved by setting the JVM option.
- For Azure DevOps Server 2020, Elastic Search 6.2.4, which ships with log4j 2.9.1, which is vulnerable and can not be resolved by setting the JVM option.
Luckily the default installation mode only binds Elastic Search to the local loopback network interface limiting access from the outside world, but if you've setup Search on a separate VM and didn't follow the Securing Search guidance to limit access you may want to look into that right now.
You currently have 3 options to deal with this issue:
- Remove Search from your installation
- Patch the log4j-core-*.jar files on disk
- Patch the log4j vulnerability at runtime
I also highly recommend you update the JVM that was installed when you configured Search. The default option does not auto-update and chances are the one you're running is already quite old.
The simplest most secure solution at the moment is to uninstall Azure DevOps Search from your Application Tiers and disabling/uninstalling the Elastic Search service:
Or remove the service altogether:
And remove the Search feature from the Azure DevOps Administration Console:
Patch Elastic Search files Instead
If you don't want to turn Search off, you'll need to patch
log4j-core to strip the vulnerable classes from the jar files. You can remove the vulnerable classes from the jar file with a single command. All of the docs I found give you this command which will work on Linux:
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
On Windows that command won't work, since the
zip command isn't available. If you've got 7-zip installed, use the following the command for Azure DevOps Server 2020:
cd "C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib" ..\bin\elasticsearch-service stop "c:\Program Files\7-Zip\7z.exe" d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class ..\bin\elasticsearch-service start
To resolve the issue for Azure DevOps Server 2019+:
cd "C:\Program Files\Azure DevOps Server 2019\Search\ES\elasticsearchv6.2\lib" ..\bin\elasticsearch-service stop "c:\Program Files\7-Zip\7z.exe" d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class ..\bin\elasticsearch-service start
To resolve the issue for Team Foundation Server 2018 Update 2+:
# cd "C:\Program Files\Microsoft Team Foundation Server 15.0\Search\ES\elasticsearch-5.4.1\lib" ..\bin\elasticsearch-service stop "c:\Program Files\7-Zip\7z.exe" d log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class ..\bin\elasticsearch-service start
Alternatively, patch log4j at runtime
You can also install
log4j-jndi-be-gone and instruct Elastic Search to load it to remove the vulnerability at runtime. This will also work for older versions of Team Foundation Server and Azure DevOps Server.
elasticsearce-service.bat manager to open the configuration panel for Elastic Search and add the jar file to the configuration:
Then restart the Elastic Search service:
Update the Java Runtime Environment for Search
If you choose to use Azul Zulu OpenJDK, ensure that you download version 8.
Extract the zip file into the
AzureDevOpsData\Java\ folder on your Application Tier: