We are happy, relieved, excited, pleased, and comforted to report that Advantage has recovered and is fully caught up processing the backlog from the past 24 hours. This includes Orders and Notes (including NAC).
Product Fulfillment is a little more nuanced and so we are taking precautions with Product fulfillment from the 5:30pm-12midnight time frame to make sure we do not over or under fulfill, and that the Advantage inventory is in sync with our warehouses. We've already captured & saved off the the product orders that need to be reviewed and The Advantage Team will continue to work on these with ACS and our warehouses. NOTE: Any NEW product fulfillment orders that came in today, and will continue to come in, will be or have already been processed. All regular Advantage PRO posting is ON.
Moving forward, reporting should not be impacted.
Huge shoutout to all the teams that immediately rallied to help solve this and fully recover in less than 24hrs. Thanks to all our clients and affiliates as well for their patience.
We will leave this incident in Monitoring Status over the weekend while we keep an eye on the health of our products and integrations. We will report back on Monday with an update.
We want to provide an update regarding the current state of the Advantage recovery efforts.
First and foremost, we want to make sure this is clear: No orders have been lost, No signups have been lost, no credit cards have been lost, No vital data has been lost.
We have made great progress, but are not 100% there yet.
- Advantage posting was resumed around 10:15am today, which means Telesales and customer service agents will be able to resume posting individual orders in their control groups, where needed. - Advantage processing queues are slowly being reopened and have already started to work through the backlog. We estimate that this will take roughly 4 hours to process and uploaded to Advantage. Please note: this is for anything from 12midnight through now. - We do still have a gap in Orders, Notes (including NAC), and Product fulfillment from appx 5:30pm– 12am last night while the servers were down. This is the focus of our recovery efforts today as we coordinate and work closely with the PIE team, very precisely and deliberately, to work through the data. We expect to be fully caught up on order processing by COB today. - NAC and Notes are impacted. We are working on the uploads and that will happen at appx. 1pm today. You may notice that some notes may be broken because the PDF letters are stored on file share (which is one of the servers we had to reboot). There is also a small risk that there might be some duplicate notes. We plan on scanning and deleting if so, but if there are any, it would be minimal.
Reminder that reporting for Zambezi, Accounting, Sales, etc.. will be impacted.
We will provide another update by COB today as we make more progress. Thank you for your patience while the teams work through the recovery.
Please do not hesitate to reach out if you have any questions or concerns. For updates in real-time, you can check here at https://14west.statuspage.io/.
Thank you, 14West Technologies Support
Posted Nov 08, 2024 - 13:03 EST
Monitoring
Most Recent Updates
A failure impacting one of the Advantage server hosts required a complete restore of several servers, including the Advantage File Share.
Critically, this file share supports WMC Order File batching for Advantage to process in OSM.
Backups are completed at 9 pm daily on these servers, which means the backup available for the restore of this file share was from November 6th, 9pm.
The Advantage team needs time tomorrow, once the servers are all back up and restored, to reassess the previous 24 hours before we can resume regular processing.
Out of an abundance of caution, we have stopped all Advantage processing as of 7 pm tonight. We will not be uploading orders, or processing credit cards, or uploading Notes to the NAC or really any other processing tomorrow until the team has had a chance to review this missing 24 hour gap. This ensures we do not accidentally cause data issues like duplicate orders or double charging credit cards.
The Advantage application UI is not down- it is UP and accessible through oneLogin, as normal. All Advantage users will still be able to go about their normal day to day functions of Customer account maintenance, Subscription maintenance, and placing orders etc..- but will not be able to right click 'Post Now'.
Reporting will only reflect through 11/7th at 7pm until we are confident to turn processing back on. This impacts Zambezi, Accounting, Sales, etc.
We will provide updates to you tomorrow as we work through this issue. Please do not hesitate to reach out if you have any questions or concerns.
Is the ability to take or process orders affected? NO
Are Customers able to access their products? YES
The Issue An isolated network event has impacted the operation of a subset of Advantage servers.
The Impact Access to the Advantage UI is currently down, while the impacted servers are being restored.
- At approximately 5:51pm US EST, an isolated network of event interrupted traffic to a host running a subset of Advantage-related servers. - Although the network issue was resolved by 6:42pm, secondary impacts to some server were discovered. - IT teams are working now to restore normal operations.
Workaround No
Next Update Updates will be posted to this incident as the affected services come back online.