Partially Degraded Service
Advantage Operational
90 days ago
100.0 % uptime
Today
WesTech Marketing Cloud ? Operational
90 days ago
100.0 % uptime
Today
Blueshift Operational
90 days ago
100.0 % uptime
Today
IRIS Operational
90 days ago
99.73 % uptime
Today
Platform Integration Engine (PIE) ? Operational
90 days ago
100.0 % uptime
Today
PIE Job API ? Operational
90 days ago
100.0 % uptime
Today
ADV Rest Adapter ? Operational
90 days ago
100.0 % uptime
Today
Hasura REST API ? Operational
90 days ago
100.0 % uptime
Today
Signup App Operational
90 days ago
100.0 % uptime
Today
Data & Analytics Operational
90 days ago
100.0 % uptime
Today
Portfolio Tracker Operational
Deliverability Operational
90 days ago
100.0 % uptime
Today
IT Operational
90 days ago
100.0 % uptime
Today
IT | General Operations Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | APJ1 ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C3-BALT ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C4-AWS-DEV ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C6-OSD ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C7-OSD ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C3 "AWS3-PRD" ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | BALT2-PRD ? Operational
90 days ago
100.0 % uptime
Today
OneLogin Operational
90 days ago
100.0 % uptime
Today
Atlassian Products Operational
90 days ago
100.0 % uptime
Today
OneTrust ? Operational
90 days ago
100.0 % uptime
Today
FinTech Degraded Performance
90 days ago
100.0 % uptime
Today
PayGate Operational
90 days ago
100.0 % uptime
Today
OneSource Operational
90 days ago
100.0 % uptime
Today
Spreedly Degraded Performance
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Past Incidents
Apr 22, 2024
Resolved - This incident has been resolved.
Apr 22, 15:40 EDT
Monitoring - OneLogin reports the issue has been resolved. We'll will continue to monitor OneLogin performance to confirm.
Apr 22, 11:17 EDT
Investigating - Is the ability to take or process orders affected?
No

Are Customers able to access their products?
Yes

The Issue
Sporadic reports of OneLogin time outs and slowness when attempting to login.

The Impact
Our team is presently investigating the true impact and severity of this incident. We are treating this as a priority.

Next Update
We will update you as soon as we have got a clearer understanding of the full impact but rest assured that we are treating this as a priority and will communicate ASAP.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support

Apr 22, 09:32 EDT
Apr 21, 2024

No incidents reported.

Apr 20, 2024

No incidents reported.

Apr 19, 2024

No incidents reported.

Apr 18, 2024

No incidents reported.

Apr 17, 2024
Resolved - Updates
The database issue has been resolved by the IT team and the upsell issues have been resolved by the WMC team by manually re-syncing the choices in our database.

Root Cause
The root cause of these issues was due to a WMC database reaching full disk space, preventing the WMC application from processing orders. This issue occured between 9:53AM (EST) - 10:52AM (EST)

Next Steps
There is existing alerting set up to alert on diskspace issues with this database, however it did not trigger. The IT team is looking into the "why" and will be updating the alerting to prevent this issue from happening again in the future. We are also working to double our available diskspace to assist with any increases of data due to regular deployments from the WMC team.

We are digging through the WMC logs to gather the list of customers that attempted to place an order but were unable to due to these issues. We will be contacting each affiliate individually with those lists today.

Please reach out to Sarah Smiley if you have any issues or concerns.

Thank you,

14West Technologies Support

Apr 17, 12:09 EDT
Update - We are continuing to monitor for any further issues.
Apr 17, 11:30 EDT
Monitoring - Is the ability to take or process orders affected?
Yes

Are Customers able to access their products?
Yes

The Issue
WMC team was alerted at 10:36 AM EST that there was an issue with our database diskspace causing errors with order processing.

* Possibly related to the database issue, Oxford and Banyan have both reported issues with loading upsell pages and placing upsell orders. The WMC team is digging into the upsell reports.

The Impact
Our team is presently investigating the true impact and severity of this incident. We are treating this as a HIGH priority. The IT team has already worked with AWS to increase diskspace and the errors related to diskspace have been resolved. WMC appears to be operating as normal however we are gathering information on impacted orders and how alerting can be improved to prevent this issue in the future.

Next Update
We will update you as soon as we have got a clearer understanding of the full impact but rest assured that we are treating this as a priority and will communicate ASAP.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support

Apr 17, 11:26 EDT
Resolved - Is the ability to take or process orders affected?
Yes

Are Customers able to access their products?
Yes

The Issue
We wanted to make you aware of an incident that happened early this morning with WMC. The team was doing a routine deployment and encountered some unexpected issues with the SQL migration that caused degraded WMC performance for 11 minutes and 30 seconds. The deployment was rolled back and the issues were fixed.

at 5:51 AM EST a database update has started
at 5:51:30 AM EST we started seeing some errors
at 6:03 AM EST- the problem is resolved, all back to normal

The Impact
Some customers may have seen pages loading slow.
9 customer pages failed to load. - We do not have data on these customers as that is captured during page load.
3 orders had been placed during this time successfully, but WMC did not render a confirmation page to these customers. - Confirmation emails were still triggered via Blueshift and sent out.

Please reach out to Sarah Smiley with any questions or concerns.

Thank you,

14West Technologies Support

Apr 17, 09:34 EDT
Apr 16, 2024
Completed - The scheduled maintenance has been completed.
Apr 16, 02:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 15, 22:00 EDT
Scheduled - ThoughtSpot service will not be available due to scheduled maintenance on Monday, April 15th beginning at 10pm EDT. This lapse in service is expected to last until 2am EDT on Tuesday, April 16th. Please reach out with any questions or concerns.

Thank you,
Data & Analytics

Apr 12, 13:10 EDT
Apr 15, 2024
Apr 14, 2024

No incidents reported.

Apr 13, 2024
Completed - The scheduled maintenance has been completed.
Apr 13, 01:30 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 12, 22:30 EDT
Scheduled - We will be conducting upgrades on various US & EU databases. The maintenance window duration will be 3 hrs (180 mins), beginning at 730pm PST on Friday, 12th April 2024, and affects both our US & EU data centers.

As a result, logging in and accessing the Blueshift US & EU platform will not be possible. This necessitates the temporary pause of ALL campaigns. While events will still be accepted, processing will be delayed. Event-triggered campaigns, especially those utilizing trigger filter segments, will be delayed. Both one-time and segment-triggered campaigns will also be impacted. Recurring campaigns scheduled during this time frame will be delayed. Syndications scheduled to go in the maintenance window will be delayed. Write APIs will be inaccessible, but READ APIs will remain available. Live Content will be unavailable, rendering "Under Maintenance" message to the user.

Upon completion of the maintenance window, campaigns will be unpaused, events will resume processing and will continue normally.

Apr 3, 19:54 EDT
Apr 12, 2024
Apr 11, 2024
Completed - The scheduled maintenance has been completed.
Apr 11, 10:30 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 11, 10:00 EDT
Scheduled - Advisory -

As a fix to the previous weekend's brief connectivity issue to SignupApp2, there will be a maintenance window to facilitate a change to the DNS records for SignupApp2 Production on Thursday, April 11 at 10:00am Eastern, lasting 30 minutes.

There is no downtime expected, nor any change to the user experience.

If you have any other questions, please feel free to reach out to your Customer Success Partner.

Thank you,
WesTech Support

Apr 9, 16:57 EDT
Apr 10, 2024

No incidents reported.

Apr 9, 2024

No incidents reported.

Apr 8, 2024
Resolved - Updates:

We feel confident in providing a resolution update on this incident.

At 12 midnight ET, SUA2 stopped processing signups. At 3.27am, we restarted the haproxy service responsible for passing traffic for the signupapp2.com domain and this resolved the issue.

The application itself has been up the whole time, but doesn’t seem to have received traffic, and the drop off in traffic seems very sudden, at 12 midnight April 7th. The incident lasted 3.5 hours, with signups processing again at 3.27am.

The customer-facing impact of this incident is that the marketing_email_subscribe event stopped working during this window, and that means that Confirmation Emails would not have fired between midnight and 3.27am ET. Signup pages were accessible throughout the incident.

Signups during that timeframe will retry and the Confirmation Email will fire once successfully processed in SUA2.

The incident is now resolved and signups are processing as expected.

Next Steps:

The incident postmortem is to be scheduled in the coming weeks after which we will share the results, as always, on Statuspage.

If you have any outstanding concerns, please reach out to your respective Customer Success Partner.

Thank you,

14West Technologies Support

Apr 8, 04:20 EDT
Investigating - Is the ability to take or process orders affected?
No
Are Customers able to access their products?
Yes
The Issue
Our teams are currently investigating reports that email signups are not being processed, and that the marketing_email_subscribe event is not firing.
The Impact
An investigation is currently underway to determine the extent of the impact. Customer-facing pages remain operational, and we don’t anticipate losing any signups. They should process once the issue is resolved.
Workaround
There is presently no workaround available.
Next Update
We are treating this as a priority and will be back ASAP with an update.
For updates in real-time, you can check here at https://14west.statuspage.io/.

Apr 8, 03:25 EDT