Log4j Detection and Response Playbook
![](https://trusted-sec.transforms.svdcdn.com/production/images/Log4j_LinkedIn_1A.jpg?w=320&h=320&auto=compress%2Cformat&fit=crop&dm=1695237755&s=bc96f78837f8617a649aa6bedd3ee8fa)
On December 09, 2021, a severe vulnerability for Apache Log4j was released (CVE-2021-44228). This vulnerability, also known as Log4Shell, allows remote code execution in many applications through web requests and without authentication. Almost immediately, many attackers on the Internet began to scan and exploit this vulnerability.
This is meant to provide guidelines and recommendations on how to prevent, detect, and mitigate this vulnerability. In general, TrustedSec recommends organizations perform the following:
- Actively scan systems or use software inventories to identify vulnerable versions of Log4J
- Update vulnerable versions of Log4j or apply mitigations
- Search for exploitation and post-exploitation activities
TrustedSec has modified sections in this blog post to accurately depict the evolving information associated with the Log4j threat. For ongoing real-time updates, it is recommended to monitor critical updates by the Cybersecurity and Infrastructure Security Agency (CISA).
1.1 Affected Versions
Log4j version 1 does not appear to be directly vulnerable, but it is end-of-life (EOL) and has not received any updates since 2015 and is also affected by separate CVEs.
The following CVEs provide additional details for the vulnerabilities associated with specific versions of Log4j:
- CVE-2021-44228 (RCE vulnerability in v2.0 through v2.14.1)
- CVE-2021-45046 (RCE vulnerability in v2.0 through v2.15.0)
- CVE-2021-4104 (RCE vulnerability in Apache Log4j 1.2)
- CVE-2021-45105 (DoS vulnerability in 2.0-beta9 to 2.16.0)
1.2 Affected Software
While Log4j is maintained by Apache, it is utilized in many vendor applications and appliances as well as in custom built systems. The following reference lists the known affected vendors as of December 12, 2021 but should not be considered definitive. Organizations should contact vendors directly for additional information.
2 Vulnerable Software Detection
Proactive scanning for vulnerable software occurs in two (2) ways: searching for vulnerable code and active scanning of deployed code.
2.1 Searching for Vulnerable Code
Searching for vulnerable code is performed by examining all servers and applications for vulnerable instances of Log4j. Because Log4j could be buried in a Java class, a simple search for Log4j will not suffice and additional methods are needed.
If clients are not currently utilizing application scanning tools, there are two (2) open-source scanning programs available to search servers and list out code versions or vulnerable code:
- Grype (https://github.com/anchore/grype) - Searches libraries installed on a system and displays vulnerabilities present
- Syft (https://github.com/anchore/syft) - Searches for installed code and libraries and displays their versions
These tools should be run on each server that is suspected of having vulnerable instances of Log4j installed.
2.1.1 Vendor Notifications
Many vendors are releasing bulletins that state if their code or appliances are vulnerable and how to apply upgrades. A list of known vulnerable vendors can be found at https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592 but should not be considered definitive. It is recommended to contact all applicable software vendors for insight into their remediation recommendations.
2.2 Active Scanning of Deployed Code
Vulnerability scanners, such as Nessus, have plugins that can actively scan a server and attempt to validate if the vulnerability exists. Public websites have also been set up to do minimal testing against an environment:
Note that vulnerability scanners typically do not check every input with a web application and may not perform authenticated scans. In-depth application assessments may be required to fully determine if a server is vulnerable.
The best project that we have seen so far to scan for the vulnerability is FullHunt's log4j-scan.
3 Prevention and Mitigation
The best way to prevent exploitation of vulnerable code is to upgrade vulnerable versions of Log4j to version 2.17.0 or apply vendor patches. Although version 2.16.0 completely removed JNDI functionality from Log4j, it was identified that Apache Log4j2 versions 2.0-alpha1 through 2.16.0, excluding 2.12.3, did not protect from uncontrolled recursion from self-referential lookups (CVE-2021-45105).
Alternatively, this infinite recursion issue can be mitigated in configuration:
- In PatternLayout in the logging configuration, replace Context Lookups like ${ctx:loginId} or $${ctx:loginId} with Thread Context Map patterns (%X, %mdc, or %MDC).
- Otherwise, in the configuration, remove references to Context Lookups like ${ctx:loginId} or $${ctx:loginId} where they originate from sources external to the application such as HTTP headers or user input.
Note that only the log4j-core JAR file is impacted by this vulnerability. Applications using only the log4j-api JAR file without the log4j-core JAR file are not impacted by this vulnerability.
If it is not possible to upgrade, there are several mitigation tactics that can be performed. However, (CVE-2021-45046) identifies additional vulnerabilities that make the flag mitigations ineffective in some cases. Therefore, the best mitigation tactic is to upgrade to 2.17.0.
- For Log4j versions >= 2.10, set the log4j2.formatMsgNoLookups system property to true on both client- and server-side components.
This can be done in multiple ways:- Add -Dlog4j2.formatMsgNoLookups=true to the startup scripts of Java programs.
- Set the following environment variable: LOG4J_FORMAT_MSG_NO_LOOKUPS=”true”.
- For Log4j versions from 2.0-beta9 through 2.10.0, remove the JndiLookup class from the classpath. For example:
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
- Isolate systems into their own restricted DMZs or VLANs.
- Block all outbound network connections from servers or limit outbound network connections to trusted hosts and network ports.
- Turn on any endpoint or network security signatures for Log4j exploitation.
- Monitor networks and servers closely for suspicious or unexpected behavior.
Testing should occur after ALL mitigation steps above have been completed to ensure that vulnerabilities do not persist.
4 Exploitation Detection
Scanning for vulnerable Log4j servers has increased significantly since the vulnerability was identified. Therefore, detection of exploitation attempts and post-exploitation activity is critical. Note that Log4j vulnerable software may be on back-end servers, so it is important for organizations to not only examine externally exposed systems but to also focus on internal systems for exploitation activity.
If any evidence of post-exploitation activity is found, Incident Response procedures should begin immediately.
4.1 Log Analysis
Exploitation attempts can be detected by searching through all log files on a server for JNDI-related exploit strings. While straight searches can be performed, attack strings can be easily obfuscated. The following script takes the obfuscation into account when searching:
- Log4Shell-detector (https://github.com/Neo23x0/log4shell-detector) - Detector for Log4Shell exploitation attempts]()
Manual searches can also be performed, but the following do not consider any obfuscation attempts. (Reference - https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b)
- Utilize the following Linux commands to search uncompressed log files in /var/log. Note that /var/log should be changed to include any additional application-specific directories.
$ sudo find /var/log/ -type f -exec sh -c "cat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -i 'jndi:(ldap[s]?|rmi|dns):'" \; $ sudo egrep -i -r '\$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+' /var/log
- Utilize the following Linux commands to search compressed log files in /var/log.
$ sudo find /var/log -name \*.gz -print0 | xargs -0 zgrep -E -i '\$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+' $ sudo find /var/log/ -name "*.gz" -type f -exec sh -c "zcat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -i 'jndi:(ldap[s]?|rmi|dns):'" \;
Additional regular expressions that can be used to detect attacks can be found on https://gist.github.com/karanlyons/8635587fd4fa5ddb4071cc44bb497ab6.
4.2 Endpoint Analysis
If an organization has visibility into the activity on their servers, endpoint post-exploitation activity can be discovered through the following methods. Note that many EDR and endpoint security systems have implemented signatures to detect this activity.
- Suspicious execution of common command line tools used to download files such as: curl, wget, or powershell
- The creation of suspicious or unexpected programs or services on an endpoint
- An increase in CPU and memory usage on a server (This is due to many attackers placing cryptominers on exploited systems.)
- Endpoint security software generating alerts for post-exploitation tool usage or activity
4.3 Network Analysis
Once an attack is successful, the victim system will often reach out to the attacker's staging server to download malicious code. Netflow data can significantly assist with the identification of this communication at the network layer or by utilizing EDR telemetry at the endpoint layer. Therefore, the following network analysis can be performed to look for post-exploitation activity:
- Search for outgoing network or web connections from your servers to the Internet.
- Outbound network connections may be to nonstandard ports or over standard HTTP/S ports.
- Look for suspicious curl or wget user agents to external IP addresses.
- Examine DNS for queries to suspicious or known malicious sites. A list of known domains and callback URLs can be found at the following links:
- Known scanning IPs - https://gist.github.com/gnremy/c546c7911d5f876f263309d7161a7217
- Known callback URLs - https://gist.github.com/superducktoes/9b742f7b44c71b4a0d19790228ce85d8
- Zeek Log4j Intelligence Feeds - https://github.com/CriticalPathSecurity/Zeek-Intelligence-Feeds/blob/master/log4j_ip.intel
5 Additional Resources
The security community has quickly come together to gather information and provide guidance for this vulnerability. Following is a list of additional resources that may also be useful:
- Log4j CVE-2021-44228 Advisory - https://github.com/advisories/GHSA-jfh8-c2jp-5v3q
- Log4j RCE Exploitation Detection - https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b
- Log4j Apache Update Site - https://logging.apache.org/log4j/2.x/download.html
- Known Affected Applications - https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592
- CrowdStrike Guidance - https://www.crowdstrike.com/blog/log4j2-vulnerability-analysis-and-mitigation-recommendations/
- Microsoft Guidance - https://www.microsoft.com/security/blog/2021/12/11/guidance-for-preventing-detecting-and-hunting-for-cve-2021-44228-log4j-2-exploitation/
- Log4j Exploitation Demonstration - https://www.youtube.com/watch?v=7qoPDq41xhQ
- Binary Defense Guidance - https://www.binarydefense.com/advice-for-defenders-responding-to-the-log4j-vulnerability-cve-2021-44228/