We are pleased to confirm that as of Friday (01/25) at 6:30 pm EST, all backed-up Workflow Event Actions had been processed after the second fix from ACS was deployed at 5:30 pm EST.
The team has been monitoring the Workflow Actions and is not seeing any indication of backlog actions. We believe the fix from ACS has fully resolved the issue.
We will be closing this incident out but if you require any further information please submit a ticket to our Support Team.
Thank you,
The Advantage Team
Posted Jan 29, 2024 - 10:10 EST
Monitoring
This morning, due to monitoring and alerting, we were made aware of a backlog of Workflow events in Advantage Production. We immediately sent to ACS Engineering Team as urgent priority and their engineers worked on a fix, which they provided to us in our Stage environment at 3:20pm today. We tested in Stage and Deployed the fix to Production at 4:30pm today. We are currently monitoring the fix to make sure the backlog drops and processes successfully. ACS has additionally provided us with a fix that will help speed up the backlog. We will deploy that to Production momentarily.
We are still working with ACS Engineering on the root cause of the issue and will report back with more details as we have them.
We will closely monitor over the weekend and provide an update on Monday.
Is the ability to take or process orders affected? No
Are Customers able to access their products? Yes
Next Update Assuming the backlog clears within the next few hours, we will provide an update on Monday with more details. If we continue to see issues and the backlog does not seem to be clearing, we will provide updates throughout the weekend.