All Systems Operational
Uptime over the past 90 days. View historical uptime.
Advantage Operational
90 days ago
100.0 % uptime
Today
WesTech Marketing Cloud ? Operational
90 days ago
100.0 % uptime
Today
Message Central Operational
90 days ago
100.0 % uptime
Today
Blueshift Operational
90 days ago
98.71 % uptime
Today
IRIS Operational
90 days ago
98.73 % uptime
Today
Opium Operational
90 days ago
100.0 % uptime
Today
Funnelcake Operational
90 days ago
100.0 % uptime
Today
Signup App Operational
90 days ago
99.61 % uptime
Today
Middleware Operational
90 days ago
100.0 % uptime
Today
Spine Operational
90 days ago
100.0 % uptime
Today
Data & Analytics Operational
90 days ago
100.0 % uptime
Today
Portfolio Tracker Operational
Deliverability Operational
90 days ago
98.73 % uptime
Today
IT Operational
90 days ago
99.99 % uptime
Today
IT | General Operations Operational
90 days ago
99.93 % uptime
Today
OpenShift Cluster | APJ1 ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C3-BALT ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C4-AWS-DEV ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C6-OSD ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C7-OSD ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C3 "AWS3-PRD" ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | BALT2-PRD ? Operational
90 days ago
100.0 % uptime
Today
OneLogin Operational
90 days ago
100.0 % uptime
Today
Atlassian Products Operational
90 days ago
100.0 % uptime
Today
OneTrust ? Operational
90 days ago
100.0 % uptime
Today
FinTech Operational
90 days ago
100.0 % uptime
Today
PayGate Operational
90 days ago
100.0 % uptime
Today
OneSource Operational
90 days ago
100.0 % uptime
Today
Spreedly Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Scheduled Maintenance
ThoughtSpot Deployment November 30th Nov 30, 2022 12:00-14:00 EST
Just a reminder that tomorrow, Wednesday November 30th, we'll be deploying Lead Gen, Marketing Performance and Lifetime Value in data hub.

The change should be seamless and likely won't be noticeable to most users but here are a few things to remember:
1. The refresh schedule will be every 2 hours.
2. Submitted signups will no longer be included in Lead Gen. They are still available in the Integrated Signup Dashboard in QlikView AND they will soon be available in ThoughtSpot via the Signup & Unsubscribe worksheet, recently released as a Beta version. Feel free to check it out while we complete the testing phase! We'll be reviewing this in upcoming office hours sessions. Actual adds will still be available in Lead Gen.
3. We're sunsetting the Lead Gen - Affiliate Level data set. Please reach out if you used it and need help transitioning to one of the other Lead Gen data sets.
Posted on Nov 29, 2022 - 15:03 EST
F5 BIGIP Code Upgrade CORP - 14.x to 15.x Feb 7, 2023 15:00-18:00 EST
The Change:

-Upgrading the F5 BIGIP load-balancer code from version 14.1.5 to 15.1.8 on our CORP appliances.

Why are we making this change?

-To keep the code as updated as possible to a supported version.

The Impact:

-There is no anticipated impact as the hardware configuration is setup as a highly available failover cluster.

What systems will this affect, and will they be available. If not, what is the timeframe for the outage?

All Corporate systems behind the F5 BIG-IP load-balancer:

These will be accessible during the maintenance:

BitBucket
Exchange - On-Prem
GlobalScape SFTP
SPINE (OpenShift)
Horizon View
Paygate (OpenShift)
QlikSense
QlikView

- Will it affect the ability to take an order? - No
- Will it affect the ability of a customer to access content? - No
- Will it affect our ability to send email? - No

The Remediation Plan:

Code will be upgraded on the standby appliance first and then a failover operation will be tested. If issues are identified, the appliance can be reverted by booting from the previous code partition. If no issues are identified, the code upgrade will proceed on the remaining appliance.
Posted on Feb 02, 2023 - 18:03 EST
Past Incidents
Feb 7, 2023

No incidents reported today.

Feb 6, 2023
ADVISORY - Internal OneLogin access
Resolved - This incident has been resolved.

We are pleased to report that a fix for the OneLogin access issue has been applied and tested successfully.

For those who might not have received the original notification, the details below are included for context:

--

Is the ability to take or process orders affected?
No

Are Customers able to access their products?
Yes

The Issue
When on an internal connection, users have been receiving a "Your connection is not private" error message

The Impact
Users on an internal network/domain connection can not reach OneLogin. Not affecting OneLogin access outside of the internal network/domain.

ETA?
Situation is actively being worked on. Resolution expected in the next hour.

Next Update
Update with then next hour.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Feb 6, 10:16 EST
Update - The issue has been identified and a fix is being implemented.

Original incident notification included below:
The Issue
When on an internal connection, users have been receiving a "Your connection is not private" error message

The Impact
Users on an internal network/domain connection can not reach OneLogin. Not affecting OneLogin access outside of the internal network/domain.
Feb 6, 10:00 EST
Identified - The issue has been identified and a fix is being implemented.
Feb 6, 09:57 EST
Update - We are continuing to investigate this issue.
Feb 6, 09:54 EST
Investigating - Is the ability to take or process orders affected?
No

Are Customers able to access their products?
Yes

The Issue
When on an internal connection, users have been receiving a "Your connection is not private" error message

The Impact
Users on an internal network/domain connection can not reach OneLogin. Not affecting OneLogin access outside of the internal network/domain.

ETA?
Situation is actively being worked on. Resolution expected in the next hour.

Next Update
Update with then next hour.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Feb 6, 09:52 EST
Feb 5, 2023

No incidents reported.

Feb 4, 2023

No incidents reported.

Feb 3, 2023
ADVISORY - Delayed Campaigns/Database Maintenance
Resolved - Updates
All campaigns and events have been unpaused now.

Root Cause
This was due to the vendor Blueshift who performed unscheduled and urgent maintenance on one of their database clusters.

Next Steps
We will continue discussions with Blueshift to understand root cause of the issue that led to the need for urgent maintenance to ensure we have limited the risk of a repeat incident.

Thank you,

14West Technologies Support
Feb 3, 08:13 EST
Update - Most Recent Updates
All campaigns have been unpaused now. However the events have been unpaused only partially, we are working on unpausing events completely.

Steps to Resolve
We have unpaused more events. The backlogs should clear in about 2 hours.

ETA/Next Update
Blueshift are addressing as a priority. We will update you as soon as all events are unpaused completely.

For updates in real-time, you can check here at https://14west.statuspage.io/ or Blueshift's StatusPage here https://status.blueshift.com.

Thank you,

14West Technologies Support
Feb 3, 06:03 EST
Monitoring - Is the ability to take or process orders affected?
No

Are Customers able to access their products?
Yes

The Issue
Blueshift had to perform an unscheduled & urgent maintenance on one of their database cluster. To operate the maintenance they had to immediately pause all campaigns from 23.02 PST.

The Impact
This is not critical, however we want to make everyone aware that this is affecting all One-Time/Recurring Campaigns as well as Segment-Triggered Campaign and Event-Triggered Campaigns. The campaigns were paused from 23.02 PST and started to be unpaused from 00.17 PST.

ETA & next update
Blueshift are addressing as a priority. We will update you as soon as all campaign were unpaused.

For updates in real-time, you can check here at https://14west.statuspage.io/ or Blueshift's StatusPage here https://status.blueshift.com.

Thank you,

14West Technologies Support
Feb 3, 04:43 EST
Feb 2, 2023

No incidents reported.

Feb 1, 2023

No incidents reported.

Jan 31, 2023
Messaging delays for campaigns
Postmortem - Read details
Feb 2, 15:24 EST
Resolved - Updates
Blueshift has informed us that the incident has now been resolved. Campaigns are sending mailings as expected.

Root Cause
We are unaware of the root cause at this time.

Next Steps
Blueshift will be providing an Incident Response in the coming days.

Thank you,

14West Technologies Support
Jan 31, 15:50 EST
Monitoring - Most Recent Updates
Blueshift informed us that they have implemented a fix and are monitoring the results.

Impact Updates
One-Time/Recurring Campaigns, Segment-Triggered Campaigns, and Event-Triggered Campaigns have been unpaused and will resume sending mailings accordingly.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support
Jan 31, 12:40 EST
Investigating - Is the ability to take or process orders affected?
No

Are Customers able to access their products?
Yes

The Issue
We have been made aware of an issue with BlueShift where messaging is delayed.

The Impact
This is affecting One-Time/Recurring Campaigns, Segment-Triggered Campaigns, and Event-Triggered Campaigns. Blueshift has informed us that they are pausing all campaigns to give the issue time to resolve

ETA & next update
Blueshift are addressing as a priority. ETA is about 2 hrs from now.

For updates in real-time, you can check here at https://14west.statuspage.io/ or Blueshift's StatusPage here https://status.blueshift.com.

Thank you,

14West Technologies Support
Jan 31, 10:35 EST
ThoughtSpot Answer and Liveboard Cleanup January 31st
Completed - The scheduled maintenance has been completed.
Jan 31, 13:00 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 31, 12:00 EST
Scheduled - In an effort to keep our ThoughtSpot environment clean, a 'clean-up' tag has been applied to any answers and liveboards that have not been used in the past 6 months. On January 31st we will be deleting anything that still has the tag applied.

Over the next two weeks please go into ThoughtSpot, select the 'clean-up' tag on the 'Answers' and 'Liveboards' tabs and review what's been flagged for deletion. If there is anything you would like to keep, simply hover over the tag for that answer/liveboard and hit the 'x'. This will remove the tag and we will not delete it.

Please note we will be doing these cleanup efforts on a quarterly basis. We'll announce at office hours and then send out a status page message so everyone has notice.

Thank you for your help,
Data and Analytics
Jan 17, 11:40 EST
Jan 30, 2023

No incidents reported.

Jan 29, 2023
IRIS Monthly System Maintenance
Completed - The scheduled maintenance has been completed.
Jan 29, 10:01 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 29, 07:00 EST
Scheduled - The IRIS system is scheduled for monthly patching on Sunday, 1/29 at 7:00 am US time and will last for approximately 3 hours. During the patching window there is no expected downtime; however the system may experience some delays processing sign-ups.

Reminders will be sent 60 mins before maintenance begins, at the start of maintenance, and then finally when maintenance is complete
Jan 26, 12:33 EST
Jan 28, 2023

No incidents reported.

Jan 27, 2023
Replacement of the SSL/TLS Certificate for the Horizon View 8
Completed - The scheduled maintenance has been completed.
Jan 27, 22:01 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 27, 21:01 EST
Update - We will be undergoing scheduled maintenance during this time.
Jan 26, 14:49 EST
Scheduled - The Change
The current SSL/TLS certificate for hrzn.14west.us (Horizon View 8) is expiring soon and needs to be replaced with a new valid certificate.

Why are we making this change?
Horizon View needs a valid SSL/TLS certificate in order to function properly and secure communication.

The Impact
Remote access into the Horizon View environment will be impacted during the maintenance window (1 hour).
New connection attempts may fail but already established connections should not be affected.

- What systems will this affect, and will they be available. If not, what is the timeframe for the outage?
Remote access into the Horizon View environment may be unavailable on 2023/01/27 - 1 hour - 9:00 pm to 10:00 pm EST
- Will it affect the ability to take an order? NO
- Will it affect the ability of a customer to access content? NO
- Will it affect our ability to send email? NO

The Remediation Plan
Import and update/replace the certificate on the Corporate F5 BigIP load-balancers.
Import and update/replace the certificate on the VMware Horizon View Connection servers.
Import and update/replace the certificate on the VMware Horizon View Unified Access Gateways.
Jan 27, 21:00 EST
Jan 26, 2023
F5 BIGIP Code Upgrade UAT - 14.x to 15.x
Completed - The scheduled maintenance has been completed.
Jan 26, 17:00 EST
Update - Code upgrade has been successfully completed.
Jan 26, 16:57 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 26, 15:00 EST
Update - UPDATED INFORMATION:

The Change
Upgrading the F5 BIGIP load-balancer code from version 14.1.5 to 15.1.8 in UAT (Non-Prod)

Why are we making this change?
To keep the code as updated as possible to a supported version.

The Impact:
There is no anticipated impact as the hardware configuration is setup as a highly available failover cluster.

- What systems will this affect, and will they be available. If not, what is the timeframe for the outage?

All Non-Prod / UAT systems behind the F5 BIG-IP load-balancer:
IRIS - UAT
OPIUM - UAT
Advantage - UAT (CardVault, Cider, API)
Middleware2 - UAT
SPINE - UAT
Qlik - UAT
SUA2 - UAT
Momentum - UAT
PayGate - UAT
OneSource - UAT (On-Prem)

- Will it affect the ability to take an order? - No
- Will it affect the ability of a customer to access content? - No
- Will it affect our ability to send email? - No

The Remediation Plan:
Code will be upgraded on the standby appliance first and then a failover operation will be tested. If issues are identified, the appliance can be reverted by booting from the previous code partition. If no issues are identified, the code upgrade will proceed on the remaining appliance.
Jan 23, 13:40 EST
Scheduled - The Change:
Upgrading the F5 BIGIP load-balancer code from version 14.1.5 to 15.1.8 in UAT (Non-Prod)

Why are we making this change?
To keep the code as updated as possible to a supported version.

The Impact:
There is no anticipated impact as the hardware configuration is setup as a highly available failover cluster.

The Remediation Plan:
Code will be upgraded on the standby appliance first and then a failover operation will be tested. If issues are identified, the appliance can be reverted by booting from the previous code partition. If no issues are identified, the code upgrade will proceed on the remaining appliance.
Jan 23, 13:08 EST
Jan 25, 2023

No incidents reported.

Jan 24, 2023

No incidents reported.

← Incident History Powered by Atlassian Statuspage