Guidance for CrowdStrike Windows Outage
Last Updated: August 1, 2024
Change Log
Date |
Updates |
|---|---|
| 8/1/2024 | 8/1/2024 - Updated What You Can Do section to include information regarding CrowdStrike deploying channel file updates utilizing enhanced procedures. |
| 7/23/2024 | Updated What CIS Is Doing and What You Can Do sections to reference the automated cloud remediation service published by CrowdStrike. |
| 7/20/2024 | Updated one more CIS action and added mention of new CrowdStrike Falcon dashboard for identifying impacted windows systems to What You Can Do section. |
| 7/19/2024 | Updates to the What Happened section to clarify the details based on input provided by CrowdStrike. Update to the What CIS is Doing section with further actions taken by CIS. |
| 7/19/2024 | Initial Publication |
What Happened
Just after midnight Eastern time July 19, Windows environments utilizing CrowdStrike Falcon software were affected when logic used in the detection capabilities of the Falcon sensor were updated for Microsoft Windows operating systems, which caused devices to experience outages. The symptoms for these outages included system crashes, Falcon sensor bugcheck screens, and Microsoft Windows blue screen error pages. Microsoft Windows systems that checked in with the CrowdStrike Cloud for updates between 0409 UTC and 0527 UTC were impacted.
NOTE: Microsoft Windows systems that checked in with the CrowdStrike Cloud for updates after 0527 UTC were not impacted as the logic update that caused the issue had been reverted at that time. CrowdStrike has since issued a workaround that requires manual remediation for each affected device.
What CIS is Doing
The CIS SOC immediately began working the issue overnight, investigating the outage and communicating with affected MS-ISAC and EI-ISAC member organizations. An email was sent from the SOC to all all CIS Endpoint Security Services / Endpoint Detection and Response customers at approximately 3 a.m. Eastern time with information about how to remediate the issue, with another update distributed at 10 a.m. A final email has been sent from the SOC to all MS- and EI-ISAC members with the most recently available guidance at 9 a.m. on July 20. Recommended actions to mitigate the impact of this issue, which were included in these correspondence, were based on official recommendations made by CrowdStrike.
On the afternoon of July 19, CIS issued a Short Form Analytic Report (SFAR) to all members of the MS- and EI-ISAC that included additional threat details related to this issue. CIS also hosted a webinar including presentations by the CIS SOC and Cyber Threat Intelligence (CTI) teams as well as representatives from CrowdStrike.
On July 23, CIS worked with CrowdStrike to enable an automated recovery for affected Windows systems for all customers of Endpoint Security Services (ESS) to assist those who are still experiencing problems recovering from this issue. If you are a customer of CIS ESS, and you have systems that still need to recover from this issue, rebooting the system should apply the fix.
We will continue to work with affected MS- and EI-ISAC members as this situation develops and as further information is made available by CrowdStrike.
What You Can Do
While remediating affected systems, organizations should be aware that CIS has detected numerous phishing campaigns and spoofed domains set up by threat actors in an attempt to socially engineer and compromise organizations affected by the outage.
CrowdStrike has widely offered remediation that utilizes the built-in quarantine functionality within the Falcon sensor to remove the problematic channel file that is causing Windows systems to crash. According to CrowdStrike, "when a Windows system with Falcon installed contacts the CrowdStrike Cloud, a request to remove the bad channel file and place it in quarantine, which is visible in your Falcon UI, will be issued. If the file does not exist, no quarantine will occur and systems will continue to operate normally." CrowdStrike also indicated that the solution could take two or three reboots once the option has been enabled to take effect. This is caused by a race condition between when Falcon quarantines the problematic file and when the problematic file is processed and activated by Falcon. For best results, CrowdStrike recommends affected systems be connected using a wired network connection to avoid latency introduced by wireless network connections. Organizations still working to recover from this issue, and that are contracted directly with CrowdStrike for services, are recommended to contact CrowdStrike Support to get set up with this remediation option.
CrowdStrike has launched a dashboard within the Falcon portal interface that will identify impacted Windows systems within your environment. This utilizes the Advanced Event Search query that was provided by CrowdStrike on July 19 for customers to identify impacted systems but makes it easier by placing it in a readily available dashboard named: hosts_possibly_impacted_by_windows_crashes. This dashboard can be found at one of the two following locations:
- Next-GEN SIEM -> or
- Investigate -> Dashboard
CrowdStrike has also provided several guides and workarounds for administrators to follow in order to recover from the issue caused by the update. These guides can be found at the CrowdStrike blog webpage linked below and cover environments such as individual physical workstations, Microsoft Azure, Amazon Web Services (AWS), Google Cloud Platform (GCP), Workspace ONE, Citrix, Rubrik, and others.
Please be advised that numerous workarounds are circulating that have not been verified. Use extreme caution when implementing anything other than the official guidance.
Update: On July 31, CrowdStrike announced that they would resume deploying channel file updates to sensors starting on August 7, 2024. Following the outage caused by a channel file update on July 19, CrowdStrike had paused the deployment of channel file updates to sensors while they investigated the root cause of the issue and used their learnings to make changes to the deployment and testing process these channel file updates utilize. Information regarding the enhancements made to the process were released in a Special Tech Alert on the CrowdStrike support portal. In addition to the enhancements made to testing and deployment processes, CrowdStrike also released options allowing customers to select how these channel file updates are applied to their sensors. These options can be selected within the General Settings menu within the CrowdStrike Falcon portal. These options are:
- Early Access – Receive the channel file update immediately following successful internal testing and deployment to CrowdStrike-controlled assets.
- General Availability (Default) – Receive the channel file update as part of a phased deployment and after successful deployment to Early Access customers. This setting is strongly recommended in the CrowdStrike Special Tech Alert and is the setting that CIS will be applying by default to those members utilizing our ESS/EDR monitoring service.
- Pause Updates – Sensors will not receive channel file updates if this option is selected. The sensor will still function properly, but will lose effectiveness over time as new sensor features or detection telemetry is released.
Official CrowdStrike guidance: https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/
We will update this page with relevant information as it becomes available.
