IRIS Custom Report Issue
Incident Report for 14 West Product
Postmortem

Incident Summary

Custom Reports in IRIS without a date range (i.e. very large reports because they’d include all dates) could not be downloaded. Attempting to do so resulted in an IRIS error page and failed download.

Leadup

There was no code change or development initiative that caused the error. There was, however, a period of maintenance that was conducted on servers used by IRIS in the 1-2 weeks prior to receiving the first Support ticket about this issue. We suspect that the accumulation of large amounts of data over time combined with an asynchronous change to a process between servers and the application resulted in this error.

Fault

The function of downloading custom reports from IRIS did not work as originally intended. This is was not the specific fault of any party, but rather the result of a large amount of historical data and server maintenance.

Impact

This issue lasted for 11 days, from October 22nd to November 2nd, and prevented stakeholders from downloading large custom reports. Every client organization in IRIS was affected, but the error only affected users trying to access a very large custom report.

Detection

This issue was raised initially through an affiliate ticket submitted to Support. Due to the isolated nature of this issue and how few users were affected, it may have been difficult to detect the error prior to an affiliate submitting a ticket.

Response

The Support, Command Center and PUMA teams responded to this issue. Once the issue was detected, there were no delays in responding to the incident and the process worked well.

Recovery

Functionality was restored by a fix implemented in IRIS code, which was deployed with release 5.20 on 10/2/20. To resolve this issue, Support informed PUMA and PUMA informed the IRIS developers at iTrans. iTrans developers quickly identified the problem and implemented a fix.

Timeline

October 21st, 7:24am EST – received the first support ticket notifying us of the issue. The Support team begins troubleshooting with the reporter.
October 22nd, 5:18am EST – received an additional support ticket. Support continues troubleshooting and reaches out to the PUMA team.
October 22nd, 12:37pm EST – PUMA sends iTrans team (who is at this time tasked on WMC) an email notifying them of the issue, however, the workday in Belarus, where iTrans is located, is over at this time.
October 23rd, 4:19pm EST – PUMA sends iTrans team a follow-up email due to a delayed response.
October 23rd, 4:31pm to 5pm EST – iTrans responds (after-hours) and clarifies priority with the Product Owner in a series of emails.
October 28th, 10:29am EST – iTrans notifies the PUMA team of the fix for the issue being deployed to the Staging environment. Several emails are changed and further discussion of the deploy to Production with an upcoming release (5.20) is determined.
November 2nd, 12:00am to 1:00am EST – The fix is deployed to Production with the IRIS v5.20 release.
November 2nd, 8:30am EST – PUMA and Support verify that the fix is functioning as designed. Errors and failed downloads when attempting to download custom reports are resolved.

Root Cause

Custom Reports in IRIS produced an error on attempted downloaded because the open date-range of a search pulled in a much larger amount of data than was ever anticipated by development.

Recurrence

This issue has not occurred before.

Lessons Learned

We learned that server maintenance with older systems can result in unanticipated OutOfMemory issues when pulling data with large date ranges in IRIS.
Communication between Support and PUMA, and initial troubleshooting conducted by Support, both went well. The resolution of the issue also went well, once priority and the deploy were determined. However, the response time of developers could be improved.

We got “lucky” in the sense that this fix could not be tested on any environment other than Production, and successfully resolved the issue on the first try.

Corrective actions

Since this issue has not ever occurred before, we do not anticipate it happening again, however, we will conduct additional testing of this feature (Custom Reports) in the future, when there is a period of server maintenance. This will be completed by the entire PUMA team and IRIS developers.

Posted Nov 04, 2020 - 13:19 EST

Resolved
The issue affecting the sporadic error with the Custom Report feature in IRIS has now been resolved. This issue was due to the application not being able to handle the large amount of data, i.e. large Custom Reports that have no date range. IRIS settings have since been changed to allow for it to be able to process more data.

BUSINESS IMPACT

You should now be able to download a Custom Report in IRIS for any date range that you choose.

For updates in real time, you can check here at https://14west.statuspage.io/.
If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center
Posted Nov 02, 2020 - 14:16 EST
Identified
As part of the new IRIS release on Monday, November 2, we will be implementing a solution to the issue affecting the Custom Report feature in IRIS. We will provide another update on our progress on Monday.

BUSINESS IMPACT

At this time, you may still be unable to download large custom reports with no date range. If you get an error when trying to download a custom report, try a smaller date range and regenerate the report.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center
Posted Oct 30, 2020 - 14:14 EDT
Update
The team is actively working on a fix for the sporadic error that occurs with Custom Report feature in IRIS. They will update you tomorrow or when the issue has been resolved.

BUSINESS IMPACT

You may still be unable to download large custom reports with no date range. If you get an error when trying to download a custom report, try a smaller date range and regenerate the report.

For updates in real time, you can check here at https://14west.statuspage.io/.
If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.
Thank you,
Global Command Center
Posted Oct 29, 2020 - 15:58 EDT
Update
The team has identified the issue that is causing the custom report feature in IRIS to occasionally error out. They are currently working on a solution and will update you tomorrow or when the issue has been resolved.

BUSINESS IMPACT

You may still be unable to download large custom reports with no date range. If you get an error when trying to download a custom report, try a smaller date range and regenerate the report.

For updates in real time, you can check here at https://14west.statuspage.io/.
If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center
Posted Oct 28, 2020 - 15:26 EDT
Update
We are still working to resolve the sporadic issue with the custom report feature in IRIS. The team will provide another update tomorrow or when the issue is resolved.
BUSINESS IMPACT
You may still be unable to download large custom reports with no date range. If you get an error when trying to download a custom report, try a smaller date range and regenerate the report.
For updates in real time, you can check here at https://14west.statuspage.io/.
If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.
Thank you,
Global Command Center
Posted Oct 27, 2020 - 16:30 EDT
Update
The team continues to investigate the sporadic issue with the custom report feature in IRIS. The team is working to resolve this issue and will provide another update tomorrow or when the issue is resolved.
BUSINESS IMPACT
You may still be unable to download large custom reports with no date range. If you get an error when trying to download a custom report, try a smaller date range and regenerate the report.
For updates in real time, you can check here at https://14west.statuspage.io/.
If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.
Thank you,
Global Command Center
Posted Oct 26, 2020 - 12:22 EDT
Update
We are continuing to investigate the intermittent issue involving the custom report feature in IRIS. The team is working to resolve this issue as quickly as possible and will provide another update on Monday or when the issue is resolved.

BUSINESS IMPACT

At this time, you may be unable to download large custom reports with no date range. If you get an error when trying to download a custom report, try a smaller date range and regenerate the report.

For updates in real time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center
Posted Oct 23, 2020 - 12:05 EDT
Investigating
We have detected an intermittent issue when attempting to download large custom reports in IRIS. At this time, you may receive a message stating maintenance is being performed when attempting to download a custom report that has a very large date range. The team is working to resolve this issue and will provide another update when we have more information or the issue is resolved.

For updates in real-time, you can check here at https://14west.statuspage.io/.

If you have any questions, concerns, or comments for the team, please email globalcommandcenter@14west.us.

Thank you,
Global Command Center
Posted Oct 22, 2020 - 13:20 EDT
This incident affected: IRIS.