Message Central Incorrect Bounce Data in Advantage

Incident Report for 14 West Product

Postmortem

Incident Summary

Bounce messages were not being populated in the message central database, which in turn meant that the bounce messages were not making it to the Advantage DB.

Leadup

No new deploys or configuration changes were deployed on the systems.

Fault

There was an extra process that was started on the MTA boxes. This process is the default out of the box log processing script. For the MC cluster, there are 2 other custom log processing scripts that populate the DB. When the default process was started it caused a conflict with the custom scripts.

Impact

Bounce data was not populated to the MC or Advantage database.

Detection

While investigating a similar issue, a member of the Deliverability team noticed that the bounce data was missing from the Advantage database.

Response

Members of the Deliverability, Command Center, Message Central, and Email Ninjas team responded to the incident as soon as it was identified.

Recovery

The issue was resolved by stopping the default process on all the MTA nodes.

The team then removed the default process from /etc/init.d stop it will not be started again.

Timeline

10/19/20 4:45 PM ET - Incident Spun Up
10/21/20 3:17 PM ET - Issue Identified
10/21/20 4:45 PM ET - Issue Resolved
10/21/20 5:40 PM ET - Resolution Communication Sent

Root Cause

Recurrence

This had never happened before.

Lessons Learned

We learned that we need to remove the default process to prevent it from being started unintentionally. The team worked well together to get the issue resolved, which further confirmed that the incident management process is effective.

Corrective actions

The Deliverability team removed the default process from /etc/init.d so that it will not be started again.

Posted Dec 11, 2020 - 10:33 EST

Resolved

The issue preventing emails marked as “Bounced” in Message Central from being reflected in Advantage has now been resolved. This issue was due to an issue with the process that writes this data to a table in Message Central.

BUSINESS IMPACT

Customers marked as “Bounced” in Message Central should now be flagged as “Bounced” in Advantage as well. We have identified some addresses that bounced and have not yet been updated in Advantage, and will be updating those records appropriately.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 21, 2020 - 17:49 EDT

Update

We are continuing to investigate why Message Central hard bounces are not being reflected in Advantage. The team is working to resolve this issue as quickly as possible and will provide another update tomorrow or when the issue is resolved.

BUSINESS IMPACT

At this time, you may not see hard bounces from Message Central in Advantage.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 20, 2020 - 16:06 EDT

Update

We have detected an issue affecting Message Central bounce data in Advantage. At this time, hard bounce events from Message Central may not be reflected in Advantage. The team is working to resolve this issue and will provide another update when we have more information or the issue is resolved.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center

Posted Oct 19, 2020 - 17:09 EDT

Investigating

We are currently investigating reports of Message Central bounce data not being accurately reflected in Advantage. We are working to resolve the issue as quickly as possible and will provide an update when we have more information.

If you have any questions please contact us at GlobalCommandCenter@14West.US.

Thank you,
Global Command Center

Posted Oct 19, 2020 - 16:50 EDT