14 West Product Status

OneTrust cookie banner script blocking Wordpress scripts incorrectly

Update - Hi all,

The Engineering and Privacy team met with OneTrust today in order to find a long term solution to this issue. At this time we are reliant on OneTrust completing an internal investigation into our Tenant and their general environment to determine why file types unrelated to cookies are being blocked upon rescan of the website and publishing of the scripts.

OneTrust's initial workaround is not a viable option for our business and we have placed a deadline of Thursday, November 21st on them coming back to us with an update.

Once we have this update, we will share it here.

Kind regards,
Privacy Team
Nov 15, 2024 - 10:04 EST

Update - The cause of the issue has been identified after a call with OneTrust on Friday night.

OneTrust's website scanner has incorrectly added file types to a block list during the latest website scans. This block list forms part of OneTrust's "Auto-Blocking" feature which if used as part of the script integration with a website will block these files until a user consents to there use. The Engineering and Privacy Team are working on removing the relevant file types from the block list, however this is a manual process and requires additional checks on .js files to ensure they contain a cookie file. This work will be completed on all impacted websites by COB tomorrow. All websites will be tested prior to the next notification.

Our OneTrust TAM will be contacted to determine why file types that are not dropping cookies have been added to the block list. Appropriate measures will be taken in the future to ensure that this issue does not arise in the future.

Thank you for your patience as this incident has been investigated.
Nov 04, 2024 - 12:45 EST

Update - Yesterday, OneTrust team made some changes to their deployment and asked us to retest the setup. Unfortunately, we still experienced issues with some of the Affiliate websites. Engineering and Privacy teams are meeting the OneTrust team today to discuss next steps.

As a temporary measure, we have advised impacted Affiliates to remove OneTrust script from their websites until we have confirmed that the issue has been resolved.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Nov 01, 2024 - 07:49 EDT

Identified - Is the ability to take or process orders affected?
NO

Are Customers able to access their products?
YES (in some cases)

The Issue
If user rejects (or partially accepts) cookie consent on the website, the website could be malfunctioning.

The Impact
The impact appears to be that images, testimonials, videos and potentially popups are not being displayed on the website until a user clicks on the “Accept All” button on the cookie banner. If a website visitor clicks on “Reject All” these images, testimonials, videos and popups are not being displayed. This will affect the user experience on impacted websites.

Workaround
If the user clicks on "Accept All" button within the cookie banner, the website will function as normal. If the user has "Rejected All" cookies before, they might need to clear their cache and history to get the cookie banner to appear again.

In some cases, where user clicks "Accept All", we're still experiencing issues with website malfunction.

Next Update
We're working with the OneTrust support on a Priority 1 ticket to get this resolved ASAP.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 31, 2024 - 06:43 EDT

ADVISORY- Blueshift SSO Cert update

Investigating - At 5:00 pm on Wednesday, November 20th, we will be updating the Blueshift SSO certificate.

We expect this to be a quick update, but you might experience a temporary error if you attempt to login while we are making this change.

Please note that any Blueshift users who are already logged in will not be affected.
Nov 14, 2024 - 08:28 EST

ADVISORY - OneLogin outage affecting US based customers

Update - We are continuing to investigate this issue.
Oct 23, 2024 - 15:40 EDT

Update - Most Recent Updates
OneLogin reports they are beginning to remediate the current issues. As their Engineers continue to restore service, to minimize downtime we encourage anyone who is currently unable to access OneLogin/Horizon to periodically try again.

Impact Updates
Until OneLogin confirms that all issues are resolved, please be aware that some may still experience issues with logging in, and that outages may reoccur.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 16, 2024 - 10:56 EDT

Investigating - Most Recent Updates
Unfortunately One Login is again experiencing issues with their portal, engineers are actively working to resolve the issue.

Impact Updates
Until OneLogin confirms that all issues are resolved, please be aware that some may still experience issues with logging in, and that outages may reoccur.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 16, 2024 - 09:27 EDT

Update - Most Recent Updates
Although OneLogin reports that their engineers are continuing their effort to restore service for all users, we've received numerous reports that recent login attempts have been successful.

Impact Updates
Until OneLogin confirms that all issues are resolved, please be aware that some may still experience issues with logging in, and that outages may reoccur.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 2024 - 16:10 EDT

Update - Most Recent Updates
OneLogin engineers are continuing their effort to restore service for all users.

Impact Updates
As OneLogin is working to remediate login issues, we are finding out that some individuals have been able to successfully access the platform. Please be aware that others may still experience issues with logging in, and that outages may return as OneLogin engineers continue to work to fully restore service.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 2024 - 15:03 EDT

Update - Most Recent Updates
OneLogin continues to be down. OneLogin engineers are working to restore service.

Impact Updates
Users accessing OneLogin may experience error pages or interruptions when trying to log into a connected application.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 2024 - 13:35 EDT

Monitoring - Most Recent Updates
OneLogin continues to be down and their engineers are working to restore service.

Impact Updates
User accessing OneLogin may experience error pages or interruptions when trying to log into a connected application.

ETA/Next Update
We will post updates from OneLogin as they arrive.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 2024 - 10:55 EDT

Investigating - Is the ability to take or process orders affected?
NO

Are Customers able to access their products?
YES

The Issue
OneLogin outage affecting US based customers.

The Impact
Attempting to authenticate to OneLogin results in errors or timeouts. Applications behind OneLogin, like Advantage, are currently inaccessible, as is remote accesses via Horizon. Applications that are locally installed on your device, like the Office Suite, should be unaffected.
OneLogin reports that they have identified the issue and their Engineers are working to restore service.

Next Update
Next update in 2 hours, or as further information is available.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Oct 15, 2024 - 09:12 EDT

CRITICAL - Intermittent Server & Workstation Outage Cause by Global Windows Anti-Virus Update Bug

Monitoring - # Most Recent Updates

Since approximately 3am US EST this morning, teams have been working to restore a fleet of Windows server and workstations which were impacted by the CrowdStrike Anti-Virus bug.

- IT has restored operations to all impacted Windows Servers.
- Product teams have confirmed all our applications (including WMC, SignuApp, and IRIS) are operating successfully.

# Steps to Resolve

For remote users with their own Windows laptops or desktop PCs, you may need to follow the instructions below to recover access to your computer:

*****

These instructions will walk you through the necessary steps to remediate your access issues due to the CrowdStrike Outage issues.

You may visit this link to access the Word document or continue reading the instructions below: https://14westit-my.sharepoint.com/:w:/r/personal/mjoebchen_theagora_com/Documents/1%20-%20Support%20Stuff/Instructions%20for%20Crowdstrike%20Outage.docx?d=w1afa9df038314a008f3b89723151a876&csf=1&web=1&e=CfVu2h

**Note, you might need to go through more than one of the options listed.

Please follow the options in the order given.

ISSUE: Blank or Black Screen

## FIRST OPTION

1. Hold down the power button for 10 seconds to turn off your device.

2. Press the power button again to turn on your device.

3. On the first sign that Windows has started (for example, some devices show the manufacturer's logo when restarting) hold down the power button for 10 seconds to turn off your device.

4. Press the power button again to turn on your device.

5. When Windows restarts, hold down the power button for 10 seconds to turn off your device.

6. Press the power button again to turn on your device.

7. When presented with Windows logon screen, if you can sign-in, do so.
**Note, if this does not resolve your issue, please proceed to Second Option***

## SECOND OPTION

1. If Step 7 reboots your device or takes you to the following screen, click on "See advanced repair option"

2. Click on Troubleshoot from "Choose an option" menu

3. Choose "Advanced Options" from "Troubleshoot' menu

4. Choose "Startup Settings" from "Advanced options" menu

5. You will need to input the BitLocker key at this stage. When entering the BitLocker key, please note the following
a. MOST IMPORTANT – you will only have about 10 seconds to start entering the BitLocker key once you get to the prompt, if you wait too long the device will reboot, so be ready please
b. You do not need to enter the " – " in between each set of 6-digits, Windows will enter that automatically
c. For most devices the Bitlocker key is:
159995-481888-409277-662277-503976-035112-345939-176297
d. If that key doesn't work, please email crowdstrike_outage@theagora.com for assistance.
e. Once you have entered the code, there is NO need to press enter, if the code is correct you will get passed to the "Startup Setting" menu (step 6).
If the code is incorrect, you will be prompted to enter the code again.
f. Even if you enter the code correctly the first time, you might need to enter a second time.

6. From the "Startup Setting" menu click "Restart". After your device restarts you will be presented with the following menu, press the number 5 key to select "Enable Safe Mode with Networking"

7. Sign into Windows using the following credentials"
a. Username – IT
b. Password – R3liabl3TacoTruck!
c. *Note: For in office workstations enter the username as: .\IT
d. If those credentials do not work, please email crowdstrike_outage@theagora.com for assistance.

8. Once logged in, you will need to navigate to the following file path
a. C:\Windows\System32\drivers\Crowdstrike
b. Locate the file starting with "C-00000291-…*sys*" and delete it.
c. To do the above, open File Explorer, in the file path type in C:\Windows\System32\drivers\Crowdstrike

9. Reboot the device and you should be good sign-in when back at the logon screen

10. For assistance with any of these steps, email crowdstrike_outage@theagora.com and provide as much detail as possible about which step failed.

*****

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Jul 19, 2024 - 12:49 EDT

Update - Summary
CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
This issue is not impacting Mac- or Linux-based hosts
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.

Current Action
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround Steps for individual hosts:
Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
Boot Windows into Safe Mode or the Windows Recovery Environment - https://support.microsoft.com/en-us/windows/start-your-pc-in-safe-mode-in-windows-92c27cff-db89-8644-1ce4-b3e5e56fe234
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.

Note: Bitlocker-encrypted hosts may require a recovery key.
Contact Miles Palmer for you Bitlocker Recovery Key - milespalmer@theagora.com
Jul 19, 2024 - 08:25 EDT

Investigating - Is the ability to take or process orders affected?
NO

Are Customers able to access their products?
YES

The Issue
All Windows Systems (servers and workstations) using CrowdStrike Anti-Virus received an automated update this morning which caused them to crash.

The Impact
Affected Windows servers or workstations are stuck in a reboot-loop, requiring manual intervention to fix.

Workaround
The teams are actively assessing and quickly remediating any crashed Windows servers or user-workstations to bring them back online.

Next Update
Regular updates to follow every 2-hours as the issue is being addressed.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Jul 19, 2024 - 07:46 EDT

Qlik Sense Refresh Delays

Identified - During a node migration for Qlik Sense, there was an issue with the scheduled refresh process. This impacted app refreshes and any pushed reports (sourced from Qlik), including the Zambezi, sent after 7:30pm ET last night.

All Qlik Sense apps are being refreshed now and we will be re-sending a corrected Zambezi report shortly.

We'll post an update once all apps have been refreshed.

Thank you,
Data & Analytics
Sep 05, 2024 - 11:18 EDT

Intermittent inaccessibility of Advantage product database.

Update - IT has investigated today’s Advantage DB outage and plan to implement a change to resolve todays Advantage Database outage. We are not 100% confident that this change will fix the issue but there is a great likely hood that it will based off our findings. This change will occur over the weekend, Sunday morning during a planned Advantage maintenance window.

In addition to the Database configuration change, IT has also setup additional monitoring to give additional insight incase this Database issue pops up again prior to this weekend’s Database configuration change.
Aug 08, 2024 - 16:05 EDT

Update - We are continuing to monitor for any further issues.
Aug 08, 2024 - 16:05 EDT

Update - We are aware of the brief Advantage outage this morning around 10:25 am EST which lasted approximately 2 minutes. The application is up and running again with all users being able to access it.

As before no orders were lost and customers could access their products.

The team are investigating this most recent product database outage and will provide an update once they have further information.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Aug 08, 2024 - 11:20 EDT

Monitoring - The teams continue to monitor the health of the Advantage platform closely. No other issues have been reported at this time.
Jul 29, 2024 - 11:05 EDT

Investigating - Is the ability to take or process orders affected?
NO

Are Customers able to access their products?
YES

The Issue

The Advantage application was briefly inaccessible for around 10 minutes from 10:30 EST.

The application is up and running with users able to access it. No orders were lost during this outage.

The IT team will continue to work on identifying a solution to prevent future outages.

The Impact
Intermittent inaccessibility of Advantage product database.

Workaround
The teams are able to restore data base health quickly, but are working on a permanent resolution.

Next Update
Updates will follow as more information is known.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Jul 16, 2024 - 11:08 EDT

Uptime over the past 90 days. View historical uptime.

Advantage Operational

90 days ago

100.0 % uptime

Today

WesTech Marketing Cloud Operational

90 days ago

100.0 % uptime

Today

Blueshift Operational

90 days ago

100.0 % uptime

Today

IRIS Operational

90 days ago

100.0 % uptime

Today

Platform Integration Engine (PIE) Operational

90 days ago

100.0 % uptime

Today

PIE Job API Operational

90 days ago

100.0 % uptime

Today

ADV Rest Adapter Operational

90 days ago

100.0 % uptime

Today

Hasura REST API Operational

90 days ago

100.0 % uptime

Today

Signup App Operational

90 days ago

100.0 % uptime

Today

Data & Analytics Operational

90 days ago

100.0 % uptime

Today

Portfolio Tracker Operational

Deliverability Operational

90 days ago

100.0 % uptime

Today

IT Operational

90 days ago

99.71 % uptime

Today

IT | General Operations Operational

90 days ago

99.94 % uptime

Today

OpenShift Cluster | APJ1 Operational

90 days ago

100.0 % uptime

Today

OpenShift Cluster | C3-BALT Operational

90 days ago

100.0 % uptime

Today

OpenShift Cluster | C4-AWS-DEV Operational

90 days ago

100.0 % uptime

Today

OpenShift Cluster | C6-OSD Operational

90 days ago

100.0 % uptime

Today

OpenShift Cluster | C7-OSD Operational

90 days ago

100.0 % uptime

Today

OpenShift Cluster | C3 "AWS3-PRD" Operational

90 days ago

100.0 % uptime

Today

OpenShift Cluster | BALT2-PRD Operational

90 days ago

100.0 % uptime

Today

OneLogin Operational

90 days ago

97.48 % uptime

Today

Atlassian Products Operational

90 days ago

100.0 % uptime

Today

OneTrust Degraded Performance

90 days ago

100.0 % uptime

Today

FinTech Degraded Performance

90 days ago

100.0 % uptime

Today

PayGate Operational

90 days ago

100.0 % uptime

Today

OneSource Operational

90 days ago

100.0 % uptime

Today

Spreedly Degraded Performance

90 days ago

100.0 % uptime

Today

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Scheduled Maintenance

IRIS Scheduled System Maintenance on 11/21 (no downtime expected) Nov 21, 2024 05:00-08:00 EST

The IRIS system is scheduled for system maintenance to begin on Thursday, 11/21 at 5:00 a.m. Eastern time, and will last for approximately 3 hours. During the maintenance window there is no expected downtime; however the system may experience some delays processing sign-ups.

Reminders will be sent 60 mins before maintenance begins, at the start of maintenance, and then finally when maintenance is complete.

Thanks,
The IRIS team
Posted on Nov 19, 2024 - 03:14 EST

Past Incidents

Nov 20, 2024

Power issues in 1001 Cathedral Street area

Resolved - BGE has fully restored power in 1001 Cathedral St.
Nov 20, 11:08 EST

Update - The 1001 building continues to have power issues.
BGE has reported an ETA of 11:15am to have the problem fixed.
Does not appear that any systems are impacted.
Nov 20, 08:52 EST

Identified - BGE has reported power issues in the 1001 Cathedral Street area.
This is causing intermittent power issues through out 1001.
Again, not all power is out at the building.
BGE has given at ETA of 11:15am for getting power fully restored.

It is not clear at this time if any systems are impacted as it appears that only the 1001 building is affected.
Nov 20, 08:38 EST

Nov 19, 2024

No incidents reported.

Nov 18, 2024

ADVISORY- Teams Working to Restore Advantage UI Access following Isolated Network Event

Resolved - This incident has been resolved.
Nov 18, 12:48 EST

Update - All,

We are happy, relieved, excited, pleased, and comforted to report that Advantage has recovered and is fully caught up processing the backlog from the past 24 hours. This includes Orders and Notes (including NAC).

Product Fulfillment is a little more nuanced and so we are taking precautions with Product fulfillment from the 5:30pm-12midnight time frame to make sure we do not over or under fulfill, and that the Advantage inventory is in sync with our warehouses. We've already captured & saved off the the product orders that need to be reviewed and The Advantage Team will continue to work on these with ACS and our warehouses. NOTE: Any NEW product fulfillment orders that came in today, and will continue to come in, will be or have already been processed. All regular Advantage PRO posting is ON.

Moving forward, reporting should not be impacted.

Huge shoutout to all the teams that immediately rallied to help solve this and fully recover in less than 24hrs. Thanks to all our clients and affiliates as well for their patience.

We will leave this incident in Monitoring Status over the weekend while we keep an eye on the health of our products and integrations. We will report back on Monday with an update.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Nov 8, 16:26 EST

Update - All,

We want to provide an update regarding the current state of the Advantage recovery efforts.

First and foremost, we want to make sure this is clear: No orders have been lost, No signups have been lost, no credit cards have been lost, No vital data has been lost.

We have made great progress, but are not 100% there yet.

- Advantage posting was resumed around 10:15am today, which means Telesales and customer service agents will be able to resume posting individual orders in their control groups, where needed.
- Advantage processing queues are slowly being reopened and have already started to work through the backlog. We estimate that this will take roughly 4 hours to process and uploaded to Advantage. Please note: this is for anything from 12midnight through now.
- We do still have a gap in Orders, Notes (including NAC), and Product fulfillment from appx 5:30pm– 12am last night while the servers were down. This is the focus of our recovery efforts today as we coordinate and work closely with the PIE team, very precisely and deliberately, to work through the data. We expect to be fully caught up on order processing by COB today.
- NAC and Notes are impacted. We are working on the uploads and that will happen at appx. 1pm today. You may notice that some notes may be broken because the PDF letters are stored on file share (which is one of the servers we had to reboot). There is also a small risk that there might be some duplicate notes. We plan on scanning and deleting if so, but if there are any, it would be minimal.

Reminder that reporting for Zambezi, Accounting, Sales, etc.. will be impacted.

We will provide another update by COB today as we make more progress. Thank you for your patience while the teams work through the recovery.

Please do not hesitate to reach out if you have any questions or concerns. For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,
14West Technologies Support
Nov 8, 13:03 EST

Monitoring - Most Recent Updates

A failure impacting one of the Advantage server hosts required a complete restore of several servers, including the Advantage File Share.

Critically, this file share supports WMC Order File batching for Advantage to process in OSM.

Backups are completed at 9 pm daily on these servers, which means the backup available for the restore of this file share was from November 6th, 9pm.

The Advantage team needs time tomorrow, once the servers are all back up and restored, to reassess the previous 24 hours before we can resume regular processing.

Out of an abundance of caution, we have stopped all Advantage processing as of 7 pm tonight. We will not be uploading orders, or processing credit cards, or uploading Notes to the NAC or really any other processing tomorrow until the team has had a chance to review this missing 24 hour gap. This ensures we do not accidentally cause data issues like duplicate orders or double charging credit cards.

The Advantage application UI is not down- it is UP and accessible through oneLogin, as normal. All Advantage users will still be able to go about their normal day to day functions of Customer account maintenance, Subscription maintenance, and placing orders etc..- but will not be able to right click 'Post Now'.

Reporting will only reflect through 11/7th at 7pm until we are confident to turn processing back on. This impacts Zambezi, Accounting, Sales, etc.

We will provide updates to you tomorrow as we work through this issue. Please do not hesitate to reach out if you have any questions or concerns.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Nov 7, 23:33 EST

Investigating - Most Recent Updates
All but one server are now back up and running.

Impact Updates
Batching will be delayed until all services are restored.

Steps to Resolve
Engineers are currently working to restore operations.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Nov 7, 20:28 EST

Identified - Is the ability to take or process orders affected?
NO

Are Customers able to access their products?
YES

The Issue
An isolated network event has impacted the operation of a subset of Advantage servers.

The Impact
Access to the Advantage UI is currently down, while the impacted servers are being restored.

- At approximately 5:51pm US EST, an isolated network of event interrupted traffic to a host running a subset of Advantage-related servers.
- Although the network issue was resolved by 6:42pm, secondary impacts to some server were discovered.
- IT teams are working now to restore normal operations.

Workaround
No

Next Update
Updates will be posted to this incident as the affected services come back online.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Nov 7, 19:40 EST

Nov 17, 2024

No incidents reported.

Nov 16, 2024

No incidents reported.

Nov 15, 2024

Unresolved incident: OneTrust cookie banner script blocking Wordpress scripts incorrectly.

Nov 14, 2024

Unresolved incident: ADVISORY- Blueshift SSO Cert update.

Nov 13, 2024

No incidents reported.

Nov 12, 2024

No incidents reported.

Nov 11, 2024

No incidents reported.

Nov 10, 2024

No incidents reported.

Nov 9, 2024

No incidents reported.

Nov 8, 2024

Nov 7, 2024

Nov 6, 2024

No incidents reported.

Related