Updates The database issue has been resolved by the IT team and the upsell issues have been resolved by the WMC team by manually re-syncing the choices in our database.
Root Cause The root cause of these issues was due to a WMC database reaching full disk space, preventing the WMC application from processing orders. This issue occured between 9:53AM (EST) - 10:52AM (EST)
Next Steps There is existing alerting set up to alert on diskspace issues with this database, however it did not trigger. The IT team is looking into the "why" and will be updating the alerting to prevent this issue from happening again in the future. We are also working to double our available diskspace to assist with any increases of data due to regular deployments from the WMC team.
We are digging through the WMC logs to gather the list of customers that attempted to place an order but were unable to due to these issues. We will be contacting each affiliate individually with those lists today.
Please reach out to Sarah Smiley if you have any issues or concerns.
Thank you,
14West Technologies Support
Posted Apr 17, 2024 - 12:09 EDT
Update
We are continuing to monitor for any further issues.
Posted Apr 17, 2024 - 11:30 EDT
Monitoring
Is the ability to take or process orders affected? Yes
Are Customers able to access their products? Yes
The Issue WMC team was alerted at 10:36 AM EST that there was an issue with our database diskspace causing errors with order processing.
* Possibly related to the database issue, Oxford and Banyan have both reported issues with loading upsell pages and placing upsell orders. The WMC team is digging into the upsell reports.
The Impact Our team is presently investigating the true impact and severity of this incident. We are treating this as a HIGH priority. The IT team has already worked with AWS to increase diskspace and the errors related to diskspace have been resolved. WMC appears to be operating as normal however we are gathering information on impacted orders and how alerting can be improved to prevent this issue in the future.
Next Update We will update you as soon as we have got a clearer understanding of the full impact but rest assured that we are treating this as a priority and will communicate ASAP.