ADVISORY - Delayed Campaigns/Database Maintenance
Incident Report for 14 West Product
Resolved
Updates
All campaigns and events have been unpaused now.

Root Cause
This was due to the vendor Blueshift who performed unscheduled and urgent maintenance on one of their database clusters.

Next Steps
We will continue discussions with Blueshift to understand root cause of the issue that led to the need for urgent maintenance to ensure we have limited the risk of a repeat incident.

Thank you,

14West Technologies Support
Posted Feb 03, 2023 - 08:13 EST
Update
Most Recent Updates
All campaigns have been unpaused now. However the events have been unpaused only partially, we are working on unpausing events completely.

Steps to Resolve
We have unpaused more events. The backlogs should clear in about 2 hours.

ETA/Next Update
Blueshift are addressing as a priority. We will update you as soon as all events are unpaused completely.

For updates in real-time, you can check here at https://14west.statuspage.io/ or Blueshift's StatusPage here https://status.blueshift.com.

Thank you,

14West Technologies Support
Posted Feb 03, 2023 - 06:03 EST
Monitoring
Is the ability to take or process orders affected?
No

Are Customers able to access their products?
Yes

The Issue
Blueshift had to perform an unscheduled & urgent maintenance on one of their database cluster. To operate the maintenance they had to immediately pause all campaigns from 23.02 PST.

The Impact
This is not critical, however we want to make everyone aware that this is affecting all One-Time/Recurring Campaigns as well as Segment-Triggered Campaign and Event-Triggered Campaigns. The campaigns were paused from 23.02 PST and started to be unpaused from 00.17 PST.

ETA & next update
Blueshift are addressing as a priority. We will update you as soon as all campaign were unpaused.

For updates in real-time, you can check here at https://14west.statuspage.io/ or Blueshift's StatusPage here https://status.blueshift.com.

Thank you,

14West Technologies Support
Posted Feb 03, 2023 - 04:43 EST
This incident affected: Blueshift.