We feel confident in providing a resolution update on this incident.
At 12 midnight ET, SUA2 stopped processing signups. At 3.27am, we restarted the haproxy service responsible for passing traffic for the signupapp2.com domain and this resolved the issue.
The application itself has been up the whole time, but doesn’t seem to have received traffic, and the drop off in traffic seems very sudden, at 12 midnight April 7th. The incident lasted 3.5 hours, with signups processing again at 3.27am.
The customer-facing impact of this incident is that the marketing_email_subscribe event stopped working during this window, and that means that Confirmation Emails would not have fired between midnight and 3.27am ET. Signup pages were accessible throughout the incident.
Signups during that timeframe will retry and the Confirmation Email will fire once successfully processed in SUA2.
The incident is now resolved and signups are processing as expected.
Next Steps:
The incident postmortem is to be scheduled in the coming weeks after which we will share the results, as always, on Statuspage.
If you have any outstanding concerns, please reach out to your respective Customer Success Partner.
Thank you,
14West Technologies Support
Posted Apr 08, 2024 - 04:20 EDT
Investigating
Is the ability to take or process orders affected? No Are Customers able to access their products? Yes The Issue Our teams are currently investigating reports that email signups are not being processed, and that the marketing_email_subscribe event is not firing. The Impact An investigation is currently underway to determine the extent of the impact. Customer-facing pages remain operational, and we don’t anticipate losing any signups. They should process once the issue is resolved. Workaround There is presently no workaround available. Next Update We are treating this as a priority and will be back ASAP with an update. For updates in real-time, you can check here at https://14west.statuspage.io/.