Partially Degraded Service
Advantage Operational
90 days ago
100.0 % uptime
Today
WesTech Marketing Cloud ? Operational
90 days ago
100.0 % uptime
Today
Blueshift Operational
90 days ago
100.0 % uptime
Today
IRIS Operational
90 days ago
99.74 % uptime
Today
Platform Integration Engine (PIE) ? Operational
90 days ago
100.0 % uptime
Today
PIE Job API ? Operational
90 days ago
100.0 % uptime
Today
ADV Rest Adapter ? Operational
90 days ago
100.0 % uptime
Today
Hasura REST API ? Operational
90 days ago
100.0 % uptime
Today
Signup App Operational
90 days ago
100.0 % uptime
Today
Data & Analytics Operational
90 days ago
100.0 % uptime
Today
Portfolio Tracker Operational
Deliverability Operational
90 days ago
100.0 % uptime
Today
IT Operational
90 days ago
100.0 % uptime
Today
IT | General Operations Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | APJ1 ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C3-BALT ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C4-AWS-DEV ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C6-OSD ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C7-OSD ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | C3 "AWS3-PRD" ? Operational
90 days ago
100.0 % uptime
Today
OpenShift Cluster | BALT2-PRD ? Operational
90 days ago
100.0 % uptime
Today
OneLogin Operational
90 days ago
100.0 % uptime
Today
Atlassian Products Operational
90 days ago
100.0 % uptime
Today
OneTrust ? Operational
90 days ago
100.0 % uptime
Today
FinTech Degraded Performance
90 days ago
100.0 % uptime
Today
PayGate Operational
90 days ago
100.0 % uptime
Today
OneSource Operational
90 days ago
100.0 % uptime
Today
Spreedly Degraded Performance
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Past Incidents
Mar 28, 2024

No incidents reported today.

Mar 27, 2024

No incidents reported.

Mar 26, 2024

No incidents reported.

Mar 25, 2024

No incidents reported.

Mar 24, 2024

No incidents reported.

Mar 23, 2024

No incidents reported.

Mar 22, 2024
Resolved - Updates
We are now 24 hours out from the restoration and we are seeing no further reports of issues. As such, we are closing down this incident.

Root Cause
We are awaiting a root cause analysis from our DNS partners at DNSMadeEasy.

Next Steps
Once we receive the above Root Cause Analysis, we will be holding our own internal postmortem in keeping with our Critical incident process. We will post this once completed.

Thank you,

14West Technologies Support

Mar 22, 18:50 EDT
Monitoring - Most Recent Updates including Impact
Our partners at DNSMadeEasy have completed the restore of our DNS back up and we are seeing a full recovery.

We will monitor over the next 24 hours to ensure it has been 100% successful.

ETA/Next Update
We will update in 24 hours once full resolution has been confirmed or earlier if we see any outstanding issues.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support

Mar 21, 20:01 EDT
Update - Most Recent Updates including Impact
The incident we've encountered today is related domains we have hosted with our DNS Partners, DNSMadeEasy. This has affected approximately 1900 domains which has made them inaccessible. Our internal Major Incident Team are presently swarming and we have so far manually fixed the 40+ most important domains. This is to ensure that the impact to orders is mitigated as best as we can. This manual resolution is allowing for us to gradually bring domains back online, which we will continue to do while we await the greater restore by DNSMadeEasy.

We can confirm that the previously mentioned issue where data in Blueshift was delayed due to the impact felt by our database being unable to complete data transfer jobs has been resolved.

We have traced back when this impact was initially felt and we can see from our logs that it occurred at approximately 1.15PM EST. Restoration attempts began at approximately 3.00PM EST.

Steps to Resolve
We continue to manually resolve domains in order of priority and will work closely with DNSMadeEasy with an aim to have the rest of the domains restored by end of day, Thursday 21st.

ETA/Next Update
We will continue to provide updates when available.

FYI - a full list of affected domains is available in our 14West Confluence space: https://14west.atlassian.net/wiki/spaces/SM/pages/47747858436/DNSMadeEasy+Incident+-+March+21st+2024

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support

Mar 21, 17:13 EDT
Identified - Most Recent Updates
We are now seeing impact on customer-facing IRIS pages and so we are updating this incident as order taking is affected. We have identified the source of the issue and have begun work with our DNS provider to resolve the issue.

Impact Updates
We also want to make you aware that there may be some impact sending of email - causing delays and soft bounces due to missing dns records. Furthermore, there may be a delay in getting up-to-date data in Blueshift due to an issue with the DB job that handles this.

Steps to Resolve
Resolution involves restoring DNS records to their original configuration. This is presently taking place. IT is first assessing any critical internal service endpoints which might have been impacted and prioritizing these.

ETA/Next Update
We are monitoring progress and will update once completed.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support

Mar 21, 14:32 EDT
Investigating - Is the ability to take or process orders affected?
No

Are Customers able to access their products?
Yes

The Issue
Our IT teams are currently investing reports of DNS resolution issues for any services operating on subdomains of 14west.io.

The Impact
We are presently investigating the level of impact but any internal systems which use subdomains of the 14west.io domain are affected. This is not affecting customers. We will provide an update on this as soon as more is known.

Workaround
There is presently no workaround available.

Next Update
We are treating this as a priority and will be back ASAP with an update.

For updates in real-time, you can check here at https://14west.statuspage.io/.

Thank you,

14West Technologies Support

Mar 21, 14:03 EDT
Completed - The scheduled maintenance has been completed.
Mar 22, 16:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 22, 14:30 EDT
Scheduled - Qlik Sense will be unavailable on Thursday, March 21st from 1pm on as we perform an upgrade. We expect it to be restored by Friday, March 22nd.
Mar 18, 14:25 EDT
Mar 21, 2024
Mar 20, 2024

No incidents reported.

Mar 19, 2024

No incidents reported.

Mar 18, 2024

No incidents reported.

Mar 17, 2024

No incidents reported.

Mar 16, 2024

No incidents reported.

Mar 15, 2024

No incidents reported.

Mar 14, 2024

No incidents reported.