Advantage Slowness Issues

Incident Report for 14 West Product

Postmortem

Incident Summary

From October 13 to October 16, some users were seeing issues when sending emails from Advantage. There was slowness, timeouts, and an error message that mentioned issues with a file path.

Leadup

There were no significant changes or other environmental factors that contributed to this incident.

Fault

Advantage was very slow and presenting a number of different errors including timeout errors. The exact cause is unknown, but it appears to have been related to Advantage’s inability to successfully access some PDF files.

Impact

From 10/13 to 10/16, users who were sending emails from Advantage were seeing slowness, timeouts, and error messages.

Detection

The issue was raised by affiliates via Support tickets. Detection could possibly have been improved by having increased monitoring on the VM, but this may have not caught these sporadic errors either.

Response

The Advantage, DBA, SysOps, and Command Center teams responded to the incident along with representatives from ACS. The process went smoothly and there were no delays that hampered the incident resolution.

Recovery

Functionality was restored on 10/16 by updating VM tools and also rebooting Maui. This lead to a gradual restoration of service.

Timeline

10/13 Tickets came in
10/13 Team members could not replicate on their own machines, via Horizon, or how TCC connects.
10/14 Maui was rebooted to help solve this issue as well as another one.
10/15 3 PM VM tools updated
10/15 5 PM update helped, until slowness was then experienced again around 5 PM from TCC. The issues tapered off and then stopped.

Root Cause

The root cause of this issue is unclear. Although the VM tools update seemed to resolve the issue, there is no other evidence that it was the root cause and not a symptom of the underlying issue.

Recurrence

This issue has not happened before.

Corrective actions

The SysOps team is looking into whether additional monitoring could help detect these types of issues in the future.

Posted Nov 09, 2020 - 17:00 EST

Resolved

The slowness issue affecting the ability to send emails in Advantage has now been resolved.

BUSINESS IMPACT

You should now be able to send emails via Advantage as you normally would.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 19, 2020 - 15:26 EDT

Update

In an effort to address the slowness or errors some users are experiencing within Advantage, we have made necessary changes to the system and are closely monitoring its progress. We will provide an update on Monday or once the issue is resolved.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 16, 2020 - 16:12 EDT

Update

The maintenance on Advantage has been completed and we are monitoring the application. We will provide another update as soon as we have more information to provide.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 15, 2020 - 16:25 EDT

Update

In an effort to resolve the slowness in Advantage, we will be performing some maintenance at 3 PM ET today. This may result in Advantage becoming unavailable for about 15 minutes. We will provide another update when the maintenance is complete.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 15, 2020 - 14:30 EDT

Update

We wanted to provide an update regarding the Advantage application server reboot. This was completed at 6:10 PM EST and we are continuing to troubleshoot any issues and will supply an update tomorrow.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 14, 2020 - 20:25 EDT

Update

While investigating the slowness issues within Advantage, we determined a reboot of the application server would be beneficial at this time. Advantage will be restarted and connection will be lost for about 15 minutes.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 14, 2020 - 17:57 EDT

Update

We are continuing to investigate slowness within Advantage. The Advantage team is working closely with ACS to resolve this issue as quickly as possible and will provide another update tomorrow when we have more information or when the issue is resolved.

BUSINESS IMPACT

At this time, you may experience issues when sending emails from Advantage.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 14, 2020 - 16:25 EDT

Investigating

We are investigating reports of slowness within Advantage. At this time, you may encounter issues when attempting to send emails from Advantage. The team is working to resolve this issue and will provide another update when we have more information or the issue is resolved.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 14, 2020 - 13:06 EDT

This incident affected: Advantage.