Partially Degraded Service
Uptime over the past 90 days. View historical uptime.
Advantage Operational
90 days ago
100.0 % uptime
Today
WesTech Marketing Cloud ? Operational
90 days ago
100.0 % uptime
Today
Blueshift Operational
90 days ago
100.0 % uptime
Today
IRIS Operational
90 days ago
100.0 % uptime
Today
Platform Integration Engine (PIE) ? Operational
90 days ago
100.0 % uptime
Today
PIE Job API ? Operational
90 days ago
100.0 % uptime
Today
ADV Rest Adapter ? Operational
90 days ago
100.0 % uptime
Today
Hasura REST API ? Operational
90 days ago
100.0 % uptime
Today
Signup App Operational
90 days ago
100.0 % uptime
Today
Data & Analytics Operational
90 days ago
100.0 % uptime
Today
Portfolio Tracker Operational
Deliverability Operational
90 days ago
100.0 % uptime
Today
IT Operational
90 days ago
99.71 % uptime
Today
IT | General Operations Operational
90 days ago
99.94 % uptime
Today
OpenShift Cluster | APJ1 ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C3-BALT ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C4-AWS-DEV ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C6-OSD ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C7-OSD ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C3 "AWS3-PRD" ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | BALT2-PRD ? Operational
90 days ago
100.0 % uptime
Today
OneLogin Operational
90 days ago
97.48 % uptime
Today
Atlassian Products Operational
90 days ago
100.0 % uptime
Today
OneTrust ? Operational
90 days ago
100.0 % uptime
Today
FinTech Degraded Performance
90 days ago
100.0 % uptime
Today
PayGate Operational
90 days ago
100.0 % uptime
Today
OneSource Operational
90 days ago
100.0 % uptime
Today
Spreedly Degraded Performance
90 days ago
100.0 % uptime
Today
Thoughtspot Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Past Incidents
Dec 9, 2024
Thoughtspot Schedule System Maintenance 12/9 (no downtime expected)
Completed - The scheduled maintenance has been completed.
Dec 9, 17:30 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 9, 17:00 EST
Scheduled - The OneLogin SSO certificate will be updated on Thoughspot on Monday 12/9 at 5pm ET.

Reminders will be sent 60 mins before maintenance begins, at the start of maintenance, and then finally when maintenance is complete.

Thanks,
I.T.
Dec 5, 13:55 EST
Dec 8, 2024

No incidents reported.

Dec 7, 2024

No incidents reported.

Dec 6, 2024
CRITICAL - Intermittent Server & Workstation Outage Cause by Global Windows Anti-Virus Update Bug
Resolved - This incident has been resolved.
Dec 6, 13:31 EST
Monitoring - # Most Recent Updates

Since approximately 3am US EST this morning, teams have been working to restore a fleet of Windows server and workstations which were impacted by the CrowdStrike Anti-Virus bug.

- IT has restored operations to all impacted Windows Servers.
- Product teams have confirmed all our applications (including WMC, SignuApp, and IRIS) are operating successfully.

# Steps to Resolve

For remote users with their own Windows laptops or desktop PCs, you may need to follow the instructions below to recover access to your computer:

*****

These instructions will walk you through the necessary steps to remediate your access issues due to the CrowdStrike Outage issues.

You may visit this link to access the Word document or continue reading the instructions below: https://14westit-my.sharepoint.com/:w:/r/personal/mjoebchen_theagora_com/Documents/1%20-%20Support%20Stuff/Instructions%20for%20Crowdstrike%20Outage.docx?d=w1afa9df038314a008f3b89723151a876&csf=1&web=1&e=CfVu2h

**Note, you might need to go through more than one of the options listed.

Please follow the options in the order given.

ISSUE: Blank or Black Screen

## FIRST OPTION

1. Hold down the power button for 10 seconds to turn off your device.

2. Press the power button again to turn on your device.

3. On the first sign that Windows has started (for example, some devices show the manufacturer's logo when restarting) hold down the power button for 10 seconds to turn off your device.

4. Press the power button again to turn on your device.

5. When Windows restarts, hold down the power button for 10 seconds to turn off your device.

6. Press the power button again to turn on your device.

7. When presented with Windows logon screen, if you can sign-in, do so.
**Note, if this does not resolve your issue, please proceed to Second Option***

## SECOND OPTION

1. If Step 7 reboots your device or takes you to the following screen, click on "See advanced repair option"

2. Click on Troubleshoot from "Choose an option" menu

3. Choose "Advanced Options" from "Troubleshoot' menu

4. Choose "Startup Settings" from "Advanced options" menu

5. You will need to input the BitLocker key at this stage. When entering the BitLocker key, please note the following
a. MOST IMPORTANT – you will only have about 10 seconds to start entering the BitLocker key once you get to the prompt, if you wait too long the device will reboot, so be ready please
b. You do not need to enter the " – " in between each set of 6-digits, Windows will enter that automatically
c. For most devices the Bitlocker key is:
159995-481888-409277-662277-503976-035112-345939-176297
d. If that key doesn't work, please email crowdstrike_outage@theagora.com for assistance.
e. Once you have entered the code, there is NO need to press enter, if the code is correct you will get passed to the "Startup Setting" menu (step 6).
If the code is incorrect, you will be prompted to enter the code again.
f. Even if you enter the code correctly the first time, you might need to enter a second time.

6. From the "Startup Setting" menu click "Restart". After your device restarts you will be presented with the following menu, press the number 5 key to select "Enable Safe Mode with Networking"

7. Sign into Windows using the following credentials"
a. Username – IT
b. Password – R3liabl3TacoTruck!
c. *Note: For in office workstations enter the username as: .\IT
d. If those credentials do not work, please email crowdstrike_outage@theagora.com for assistance.

8. Once logged in, you will need to navigate to the following file path
a. C:\Windows\System32\drivers\Crowdstrike
b. Locate the file starting with "C-00000291-…*sys*" and delete it.
c. To do the above, open File Explorer, in the file path type in C:\Windows\System32\drivers\Crowdstrike

9. Reboot the device and you should be good sign-in when back at the logon screen

10. For assistance with any of these steps, email crowdstrike_outage@theagora.com and provide as much detail as possible about which step failed.

*****

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Jul 19, 12:49 EDT
Update - Summary
CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
This issue is not impacting Mac- or Linux-based hosts
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.

Current Action
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround Steps for individual hosts:
Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
Boot Windows into Safe Mode or the Windows Recovery Environment - https://support.microsoft.com/en-us/windows/start-your-pc-in-safe-mode-in-windows-92c27cff-db89-8644-1ce4-b3e5e56fe234
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.

Note: Bitlocker-encrypted hosts may require a recovery key.
Contact Miles Palmer for you Bitlocker Recovery Key - milespalmer@theagora.com
Jul 19, 08:25 EDT
Investigating - Is the ability to take or process orders affected?
NO

Are Customers able to access their products?
YES

The Issue
All Windows Systems (servers and workstations) using CrowdStrike Anti-Virus received an automated update this morning which caused them to crash.

The Impact
Affected Windows servers or workstations are stuck in a reboot-loop, requiring manual intervention to fix.

Workaround
The teams are actively assessing and quickly remediating any crashed Windows servers or user-workstations to bring them back online.

Next Update
Regular updates to follow every 2-hours as the issue is being addressed.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Jul 19, 07:46 EDT
Intermittent inaccessibility of Advantage product database.
Resolved - This incident has been resolved.
Dec 6, 10:39 EST
Update - IT has investigated today’s Advantage DB outage and plan to implement a change to resolve todays Advantage Database outage. We are not 100% confident that this change will fix the issue but there is a great likely hood that it will based off our findings. This change will occur over the weekend, Sunday morning during a planned Advantage maintenance window.

In addition to the Database configuration change, IT has also setup additional monitoring to give additional insight incase this Database issue pops up again prior to this weekend’s Database configuration change.
Aug 8, 16:05 EDT
Update - We are continuing to monitor for any further issues.
Aug 8, 16:05 EDT
Update - We are aware of the brief Advantage outage this morning around 10:25 am EST which lasted approximately 2 minutes. The application is up and running again with all users being able to access it.

As before no orders were lost and customers could access their products.

The team are investigating this most recent product database outage and will provide an update once they have further information.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Aug 8, 11:20 EDT
Monitoring - The teams continue to monitor the health of the Advantage platform closely. No other issues have been reported at this time.
Jul 29, 11:05 EDT
Investigating - Is the ability to take or process orders affected?
NO

Are Customers able to access their products?
YES

The Issue

The Advantage application was briefly inaccessible for around 10 minutes from 10:30 EST.

The application is up and running with users able to access it. No orders were lost during this outage.

The IT team will continue to work on identifying a solution to prevent future outages.

The Impact
Intermittent inaccessibility of Advantage product database.

Workaround
The teams are able to restore data base health quickly, but are working on a permanent resolution.

Next Update
Updates will follow as more information is known.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Jul 16, 11:08 EDT
ADVISORY - OneLogin outage affecting US based customers
Resolved - This incident has been resolved.
Dec 6, 10:38 EST
Update - We are continuing to investigate this issue.
Oct 23, 15:40 EDT
Update - Most Recent Updates
OneLogin reports they are beginning to remediate the current issues. As their Engineers continue to restore service, to minimize downtime we encourage anyone who is currently unable to access OneLogin/Horizon to periodically try again.

Impact Updates
Until OneLogin confirms that all issues are resolved, please be aware that some may still experience issues with logging in, and that outages may reoccur.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 16, 10:56 EDT
Investigating - Most Recent Updates
Unfortunately One Login is again experiencing issues with their portal, engineers are actively working to resolve the issue.

Impact Updates
Until OneLogin confirms that all issues are resolved, please be aware that some may still experience issues with logging in, and that outages may reoccur.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 16, 09:27 EDT
Update - Most Recent Updates
Although OneLogin reports that their engineers are continuing their effort to restore service for all users, we've received numerous reports that recent login attempts have been successful.

Impact Updates
Until OneLogin confirms that all issues are resolved, please be aware that some may still experience issues with logging in, and that outages may reoccur.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 16:10 EDT
Update - Most Recent Updates
OneLogin engineers are continuing their effort to restore service for all users.

Impact Updates
As OneLogin is working to remediate login issues, we are finding out that some individuals have been able to successfully access the platform. Please be aware that others may still experience issues with logging in, and that outages may return as OneLogin engineers continue to work to fully restore service.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 15:03 EDT
Update - Most Recent Updates
OneLogin continues to be down. OneLogin engineers are working to restore service.

Impact Updates
Users accessing OneLogin may experience error pages or interruptions when trying to log into a connected application.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 13:35 EDT
Update - Most Recent Updates
OneLogin continues to be down. OneLogin engineers are working to restore service.

Impact Updates
Users accessing OneLogin may experience error pages or interruptions when trying to log into a connected application.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 12:00 EDT
Monitoring - Most Recent Updates
OneLogin continues to be down and their engineers are working to restore service.

Impact Updates
User accessing OneLogin may experience error pages or interruptions when trying to log into a connected application.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 10:55 EDT
Investigating - Is the ability to take or process orders affected?
NO

Are Customers able to access their products?
YES

The Issue
OneLogin outage affecting US based customers.

The Impact
Attempting to authenticate to OneLogin results in errors or timeouts. Applications behind OneLogin, like Advantage, are currently inaccessible, as is remote accesses via Horizon. Applications that are locally installed on your device, like the Office Suite, should be unaffected.
OneLogin reports that they have identified the issue and their Engineers are working to restore service.

Next Update
Next update in 2 hours, or as further information is available.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 09:12 EDT
Qlik Sense Refresh Delays
Resolved - This incident has been resolved.
Dec 6, 09:54 EST
Identified - During a node migration for Qlik Sense, there was an issue with the scheduled refresh process. This impacted app refreshes and any pushed reports (sourced from Qlik), including the Zambezi, sent after 7:30pm ET last night.

All Qlik Sense apps are being refreshed now and we will be re-sending a corrected Zambezi report shortly.

We'll post an update once all apps have been refreshed.

Thank you,
Data & Analytics
Sep 5, 11:18 EDT
Dec 5, 2024

No incidents reported.

Dec 4, 2024

No incidents reported.

Dec 3, 2024

No incidents reported.

Dec 2, 2024

No incidents reported.

Dec 1, 2024

No incidents reported.

Nov 30, 2024

No incidents reported.

Nov 29, 2024

No incidents reported.

Nov 28, 2024

No incidents reported.

Nov 27, 2024
Delayed Signups Overnight
Resolved - There were lingering issues from the Hasura outage yesterday that caused a delay in signup processing. We were able to identify the issue and we restarted processing around 8:30am Eastern today. All delayed signups have been processed and we are continuing to monitor and ensure that signups are processing as expected. No signups were lost, and reporting should be caught up to reflect accurate counts from the overnight signups.
-14 West Technology
Nov 27, 09:30 EST
Nov 26, 2024
Brief Hasura outage causing failures when submitting Signups and processing Login requests
Resolved - As a result of a Hasura outage caused by issues in our AWS project, some PIE calls were impacted and returned error messages. We performed a Restart of the backend of the PIE applications to refresh the connection pool, which has resolved the issues. The outage clogged these connections, so the Restart cleared them and got us back on track. The signups collected during this time have just finished processing, and the applications appear to be working as expected. We are continually monitoring the situation and will ensure that our applications continue to function properly. Thankfully, due to safeguards that we have built, no signups were lost.
-14 West Technology
Hasura incident details may be found at https://status.hasura.io/incidents/8kdg3cjv2vzq
Nov 26, 12:00 EST
Nov 25, 2024

No incidents reported.

← Incident History Powered by Atlassian Statuspage