Monitoring - # Most Recent Updates
Since approximately 3am US EST this morning, teams have been working to restore a fleet of Windows server and workstations which were impacted by the CrowdStrike Anti-Virus bug.
- IT has restored operations to all impacted Windows Servers.
- Product teams have confirmed all our applications (including WMC, SignuApp, and IRIS) are operating successfully.
# Steps to Resolve
For remote users with their own Windows laptops or desktop PCs, you may need to follow the instructions below to recover access to your computer:
*****
These instructions will walk you through the necessary steps to remediate your access issues due to the CrowdStrike Outage issues.
You may visit this link to access the Word document or continue reading the instructions below: https://14westit-my.sharepoint.com/:w:/r/personal/mjoebchen_theagora_com/Documents/1%20-%20Support%20Stuff/Instructions%20for%20Crowdstrike%20Outage.docx?d=w1afa9df038314a008f3b89723151a876&csf=1&web=1&e=CfVu2h
**Note, you might need to go through more than one of the options listed.
Please follow the options in the order given.
ISSUE: Blank or Black Screen
## FIRST OPTION
1. Hold down the power button for 10 seconds to turn off your device.
2. Press the power button again to turn on your device.
3. On the first sign that Windows has started (for example, some devices show the manufacturer's logo when restarting) hold down the power button for 10 seconds to turn off your device.
4. Press the power button again to turn on your device.
5. When Windows restarts, hold down the power button for 10 seconds to turn off your device.
6. Press the power button again to turn on your device.
7. When presented with Windows logon screen, if you can sign-in, do so.
**Note, if this does not resolve your issue, please proceed to Second Option***
## SECOND OPTION
1. If Step 7 reboots your device or takes you to the following screen, click on "See advanced repair option"
2. Click on Troubleshoot from "Choose an option" menu
3. Choose "Advanced Options" from "Troubleshoot' menu
4. Choose "Startup Settings" from "Advanced options" menu
5. You will need to input the BitLocker key at this stage. When entering the BitLocker key, please note the following
a. MOST IMPORTANT – you will only have about 10 seconds to start entering the BitLocker key once you get to the prompt, if you wait too long the device will reboot, so be ready please
b. You do not need to enter the " – " in between each set of 6-digits, Windows will enter that automatically
c. For most devices the Bitlocker key is:
159995-481888-409277-662277-503976-035112-345939-176297
d. If that key doesn't work, please email crowdstrike_outage@theagora.com for assistance.
e. Once you have entered the code, there is NO need to press enter, if the code is correct you will get passed to the "Startup Setting" menu (step 6).
If the code is incorrect, you will be prompted to enter the code again.
f. Even if you enter the code correctly the first time, you might need to enter a second time.
6. From the "Startup Setting" menu click "Restart". After your device restarts you will be presented with the following menu, press the number 5 key to select "Enable Safe Mode with Networking"
7. Sign into Windows using the following credentials"
a. Username – IT
b. Password – R3liabl3TacoTruck!
c. *Note: For in office workstations enter the username as: .\IT
d. If those credentials do not work, please email crowdstrike_outage@theagora.com for assistance.
8. Once logged in, you will need to navigate to the following file path
a. C:\Windows\System32\drivers\Crowdstrike
b. Locate the file starting with "C-00000291-…*sys*" and delete it.
c. To do the above, open File Explorer, in the file path type in C:\Windows\System32\drivers\Crowdstrike
9. Reboot the device and you should be good sign-in when back at the logon screen
10. For assistance with any of these steps, email crowdstrike_outage@theagora.com and provide as much detail as possible about which step failed.
*****
For updates in real-time, you can check here at https://14west.statuspage.io/.
Thank you,
14West Technologies Support
Jul 19, 2024 - 12:49 EDT
Since approximately 3am US EST this morning, teams have been working to restore a fleet of Windows server and workstations which were impacted by the CrowdStrike Anti-Virus bug.
- IT has restored operations to all impacted Windows Servers.
- Product teams have confirmed all our applications (including WMC, SignuApp, and IRIS) are operating successfully.
# Steps to Resolve
For remote users with their own Windows laptops or desktop PCs, you may need to follow the instructions below to recover access to your computer:
*****
These instructions will walk you through the necessary steps to remediate your access issues due to the CrowdStrike Outage issues.
You may visit this link to access the Word document or continue reading the instructions below: https://14westit-my.sharepoint.com/:w:/r/personal/mjoebchen_theagora_com/Documents/1%20-%20Support%20Stuff/Instructions%20for%20Crowdstrike%20Outage.docx?d=w1afa9df038314a008f3b89723151a876&csf=1&web=1&e=CfVu2h
**Note, you might need to go through more than one of the options listed.
Please follow the options in the order given.
ISSUE: Blank or Black Screen
## FIRST OPTION
1. Hold down the power button for 10 seconds to turn off your device.
2. Press the power button again to turn on your device.
3. On the first sign that Windows has started (for example, some devices show the manufacturer's logo when restarting) hold down the power button for 10 seconds to turn off your device.
4. Press the power button again to turn on your device.
5. When Windows restarts, hold down the power button for 10 seconds to turn off your device.
6. Press the power button again to turn on your device.
7. When presented with Windows logon screen, if you can sign-in, do so.
**Note, if this does not resolve your issue, please proceed to Second Option***
## SECOND OPTION
1. If Step 7 reboots your device or takes you to the following screen, click on "See advanced repair option"
2. Click on Troubleshoot from "Choose an option" menu
3. Choose "Advanced Options" from "Troubleshoot' menu
4. Choose "Startup Settings" from "Advanced options" menu
5. You will need to input the BitLocker key at this stage. When entering the BitLocker key, please note the following
a. MOST IMPORTANT – you will only have about 10 seconds to start entering the BitLocker key once you get to the prompt, if you wait too long the device will reboot, so be ready please
b. You do not need to enter the " – " in between each set of 6-digits, Windows will enter that automatically
c. For most devices the Bitlocker key is:
159995-481888-409277-662277-503976-035112-345939-176297
d. If that key doesn't work, please email crowdstrike_outage@theagora.com for assistance.
e. Once you have entered the code, there is NO need to press enter, if the code is correct you will get passed to the "Startup Setting" menu (step 6).
If the code is incorrect, you will be prompted to enter the code again.
f. Even if you enter the code correctly the first time, you might need to enter a second time.
6. From the "Startup Setting" menu click "Restart". After your device restarts you will be presented with the following menu, press the number 5 key to select "Enable Safe Mode with Networking"
7. Sign into Windows using the following credentials"
a. Username – IT
b. Password – R3liabl3TacoTruck!
c. *Note: For in office workstations enter the username as: .\IT
d. If those credentials do not work, please email crowdstrike_outage@theagora.com for assistance.
8. Once logged in, you will need to navigate to the following file path
a. C:\Windows\System32\drivers\Crowdstrike
b. Locate the file starting with "C-00000291-…*sys*" and delete it.
c. To do the above, open File Explorer, in the file path type in C:\Windows\System32\drivers\Crowdstrike
9. Reboot the device and you should be good sign-in when back at the logon screen
10. For assistance with any of these steps, email crowdstrike_outage@theagora.com and provide as much detail as possible about which step failed.
*****
For updates in real-time, you can check here at https://14west.statuspage.io/.
Thank you,
14West Technologies Support
Jul 19, 2024 - 12:49 EDT
Update - Summary
CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.
Details
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
This issue is not impacting Mac- or Linux-based hosts
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
Current Action
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:
Workaround Steps for individual hosts:
Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
Boot Windows into Safe Mode or the Windows Recovery Environment - https://support.microsoft.com/en-us/windows/start-your-pc-in-safe-mode-in-windows-92c27cff-db89-8644-1ce4-b3e5e56fe234
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
Note: Bitlocker-encrypted hosts may require a recovery key.
Contact Miles Palmer for you Bitlocker Recovery Key - milespalmer@theagora.com
Jul 19, 2024 - 08:25 EDT
CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.
Details
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
This issue is not impacting Mac- or Linux-based hosts
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
Current Action
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:
Workaround Steps for individual hosts:
Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
Boot Windows into Safe Mode or the Windows Recovery Environment - https://support.microsoft.com/en-us/windows/start-your-pc-in-safe-mode-in-windows-92c27cff-db89-8644-1ce4-b3e5e56fe234
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
Note: Bitlocker-encrypted hosts may require a recovery key.
Contact Miles Palmer for you Bitlocker Recovery Key - milespalmer@theagora.com
Jul 19, 2024 - 08:25 EDT
Investigating - Is the ability to take or process orders affected?
NO
Are Customers able to access their products?
YES
The Issue
All Windows Systems (servers and workstations) using CrowdStrike Anti-Virus received an automated update this morning which caused them to crash.
The Impact
Affected Windows servers or workstations are stuck in a reboot-loop, requiring manual intervention to fix.
Workaround
The teams are actively assessing and quickly remediating any crashed Windows servers or user-workstations to bring them back online.
Next Update
Regular updates to follow every 2-hours as the issue is being addressed.
For updates in real-time, you can check here at https://14west.statuspage.io/.
Thank you,
14West Technologies Support
Jul 19, 2024 - 07:46 EDT
NO
Are Customers able to access their products?
YES
The Issue
All Windows Systems (servers and workstations) using CrowdStrike Anti-Virus received an automated update this morning which caused them to crash.
The Impact
Affected Windows servers or workstations are stuck in a reboot-loop, requiring manual intervention to fix.
Workaround
The teams are actively assessing and quickly remediating any crashed Windows servers or user-workstations to bring them back online.
Next Update
Regular updates to follow every 2-hours as the issue is being addressed.
For updates in real-time, you can check here at https://14west.statuspage.io/.
Thank you,
14West Technologies Support
Jul 19, 2024 - 07:46 EDT
Investigating - Is the ability to take or process orders affected?
NO
Are Customers able to access their products?
YES
The Issue
The Advantage application was briefly inaccessible for around 10 minutes from 10:30 EST.
The application is up and running with users able to access it. No orders were lost during this outage.
The IT team will continue to work on identifying a solution to prevent future outages.
The Impact
Intermittent inaccessibility of Advantage product database.
Workaround
The teams are able to restore data base health quickly, but are working on a permanent resolution.
Next Update
Updates will follow as more information is known.
For updates in real-time, you can check here at https://14west.statuspage.io/.
Thank you,
14West Technologies Support
Jul 16, 2024 - 11:08 EDT
NO
Are Customers able to access their products?
YES
The Issue
The Advantage application was briefly inaccessible for around 10 minutes from 10:30 EST.
The application is up and running with users able to access it. No orders were lost during this outage.
The IT team will continue to work on identifying a solution to prevent future outages.
The Impact
Intermittent inaccessibility of Advantage product database.
Workaround
The teams are able to restore data base health quickly, but are working on a permanent resolution.
Next Update
Updates will follow as more information is known.
For updates in real-time, you can check here at https://14west.statuspage.io/.
Thank you,
14West Technologies Support
Jul 16, 2024 - 11:08 EDT
Advantage
Operational
WesTech Marketing Cloud
?
Operational
Blueshift
Operational
IRIS
Operational
Platform Integration Engine (PIE)
?
Operational
PIE Job API
?
Operational
ADV Rest Adapter
?
Operational
Hasura REST API
?
Operational
Signup App
Operational
Data & Analytics
Operational
Portfolio Tracker
Operational
Deliverability
Operational
IT
Operational
IT | General Operations
Operational
OpenShift Cluster | APJ1
?
Operational
OpenShift Cluster | C3-BALT
?
Operational
OpenShift Cluster | C4-AWS-DEV
?
Operational
OpenShift Cluster | C6-OSD
?
Operational
OpenShift Cluster | C7-OSD
?
Operational
OpenShift Cluster | C3 "AWS3-PRD"
?
Operational
OpenShift Cluster | BALT2-PRD
?
Operational
OneLogin
Operational
Atlassian Products
Operational
OneTrust
?
Operational
FinTech
Degraded Performance
PayGate
Operational
OneSource
Operational
Spreedly
Degraded Performance
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Scheduled Maintenance
Horizon Maintenance
Jul 28, 2024 00:00-01:00 EDT
We will be replacing the Horizon certificate. During this time Horizon will not allow connections or apps to launch.
Posted on Jul 24, 2024 - 11:13 EDT
Posted on Jul 24, 2024 - 11:13 EDT
IRIS Scheduled System Maintenance on 7/28 (no downtime expected)
Jul 28, 2024 08:00-11:00 EDT
The IRIS system is scheduled for system maintenance to begin on Sunday, 7/28 at 8:00 a.m. US time, and will last for approximately 3 hours. During the maintenance window there is no expected downtime; however the system may experience some delays processing sign-ups. Reminders will be sent 60 mins before maintenance begins, at the start of maintenance, and then finally when maintenance is complete.
Thanks,
The IRIS team
Posted on Jul 28, 2024 - 04:50 EDT
Thanks,
The IRIS team
Posted on Jul 28, 2024 - 04:50 EDT