Since approximately 3am US EST this morning, teams have been working to restore a fleet of Windows server and workstations which were impacted by the CrowdStrike Anti-Virus bug.
- IT has restored operations to all impacted Windows Servers. - Product teams have confirmed all our applications (including WMC, SignuApp, and IRIS) are operating successfully.
# Steps to Resolve
For remote users with their own Windows laptops or desktop PCs, you may need to follow the instructions below to recover access to your computer:
*****
These instructions will walk you through the necessary steps to remediate your access issues due to the CrowdStrike Outage issues.
**Note, you might need to go through more than one of the options listed.
Please follow the options in the order given.
ISSUE: Blank or Black Screen
## FIRST OPTION
1. Hold down the power button for 10 seconds to turn off your device.
2. Press the power button again to turn on your device.
3. On the first sign that Windows has started (for example, some devices show the manufacturer's logo when restarting) hold down the power button for 10 seconds to turn off your device.
4. Press the power button again to turn on your device.
5. When Windows restarts, hold down the power button for 10 seconds to turn off your device.
6. Press the power button again to turn on your device.
7. When presented with Windows logon screen, if you can sign-in, do so. **Note, if this does not resolve your issue, please proceed to Second Option***
## SECOND OPTION
1. If Step 7 reboots your device or takes you to the following screen, click on "See advanced repair option"
2. Click on Troubleshoot from "Choose an option" menu
3. Choose "Advanced Options" from "Troubleshoot' menu
4. Choose "Startup Settings" from "Advanced options" menu
5. You will need to input the BitLocker key at this stage. When entering the BitLocker key, please note the following a. MOST IMPORTANT – you will only have about 10 seconds to start entering the BitLocker key once you get to the prompt, if you wait too long the device will reboot, so be ready please b. You do not need to enter the " – " in between each set of 6-digits, Windows will enter that automatically c. For most devices the Bitlocker key is: 159995-481888-409277-662277-503976-035112-345939-176297 d. If that key doesn't work, please email crowdstrike_outage@theagora.com for assistance. e. Once you have entered the code, there is NO need to press enter, if the code is correct you will get passed to the "Startup Setting" menu (step 6). If the code is incorrect, you will be prompted to enter the code again. f. Even if you enter the code correctly the first time, you might need to enter a second time.
6. From the "Startup Setting" menu click "Restart". After your device restarts you will be presented with the following menu, press the number 5 key to select "Enable Safe Mode with Networking"
7. Sign into Windows using the following credentials" a. Username – IT b. Password – R3liabl3TacoTruck! c. *Note: For in office workstations enter the username as: .\IT d. If those credentials do not work, please email crowdstrike_outage@theagora.com for assistance.
8. Once logged in, you will need to navigate to the following file path a. C:\Windows\System32\drivers\Crowdstrike b. Locate the file starting with "C-00000291-…*sys*" and delete it. c. To do the above, open File Explorer, in the file path type in C:\Windows\System32\drivers\Crowdstrike
9. Reboot the device and you should be good sign-in when back at the logon screen
10. For assistance with any of these steps, email crowdstrike_outage@theagora.com and provide as much detail as possible about which step failed.
Summary CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.
Details Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor. This issue is not impacting Mac- or Linux-based hosts Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
Current Action CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes. If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:
Workaround Steps for individual hosts: Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then: Boot Windows into Safe Mode or the Windows Recovery Environment - https://support.microsoft.com/en-us/windows/start-your-pc-in-safe-mode-in-windows-92c27cff-db89-8644-1ce4-b3e5e56fe234 Navigate to the C:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Boot the host normally.
Note: Bitlocker-encrypted hosts may require a recovery key. Contact Miles Palmer for you Bitlocker Recovery Key - milespalmer@theagora.com
Posted Jul 19, 2024 - 08:25 EDT
Investigating
Is the ability to take or process orders affected? NO
Are Customers able to access their products? YES
The Issue All Windows Systems (servers and workstations) using CrowdStrike Anti-Virus received an automated update this morning which caused them to crash.
The Impact Affected Windows servers or workstations are stuck in a reboot-loop, requiring manual intervention to fix.
Workaround The teams are actively assessing and quickly remediating any crashed Windows servers or user-workstations to bring them back online.
Next Update Regular updates to follow every 2-hours as the issue is being addressed.