UPDATE--An issue with an update for versions of CrowdStrike’s Falcon EDR product running on Windows machines late Thursday night caused those machines to fail and go into a boot loop state, causing widespread issues and a cascading series of outages for companies and services across the Internet.
Many banks, airlines, media companies, and other large entities were forced to take services offline overnight, with airlines including United, Delta, and others grounding all of their flights for hours. The outages are not connected to an attack, but rather the result of a faulty update that CrowdStrike pushed Thursday night. That update appears to have affected machines running in Microsoft’s Azure cloud platform, as well, which serves as the hosting platform for a huge number of organizations around the world.
“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website,” CrowdStrike CEO George Kurtz said.
Microsoft estimates that about 8.5 million Windows machines were affected by the CrowdStrike update, and the company has released a recovery tool to help IT administrators recover from the outage.
"While software updates may occasionally cause disturbances, significant incidents like the CrowdStrike event are infrequent. We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices, or less than one percent of all Windows machines. While the percentage was small, the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services," David Weston, vice president of enterprise and OS security at Microsoft, said in a post Saturday.
CrowdStrike is one of the larger providers of endpoint and server security products in the world and has customers in virtually every major industry, including finance, travel, technology, government, and many more. Many airlines and other companies have posted notices on their apps and websites about the issue.
“A third-party software outage impacted computer systems worldwide, including at United. We are resuming some flights but expect schedule disruptions to continue throughout Friday,” United Airlines said on its app.
CrowdStrike is recommending that organizations take the following steps to remediate any Windows machines affected by the update:
Boot Windows into Safe Mode or the Windows Recovery Environment Navigate to the C:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Boot the host normally.
Microsoft said on its Azure status page that the problem began around 19:00 UTC on July 18.
“We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD) and get stuck in a restarting state,” the status page says.
This story was updated on July 20 to add information from Microsoft.