Label printing and downloading issue

Incident Report for nShift - Transsmart

Postmortem

At Tuesday 2-Apr-2024 around 15:40 CET downloading labels stopped working. There were connection issues in the MyTranssmart dashboard when trying to download labels, resulting in timeouts. As a result, the printing service (which also handles the downloads) got overwhelmed and couldn't handle all print requests in time anymore. During investigation, we found out Windows updates were done on a production server that hosted the label generation application. The other server was already updated some weeks ago, causing no issues, so it's not clear why the second server now caused issues. Rebooting that server solved the connection issues.

Timings: At 16:40 we restarted the API printing services. Printing labels worked again. Downloading labels still wasn't stable; it didn't always work. At 19:50 we restarted the server on which Windows updates were done. Downloading labels worked again.

We apologize for the inconvenience it may have caused. We will improve the internal communication channels between our shared services accordingly and will add extra monitoring to the label generation application.

Posted Apr 03, 2024 - 11:25 CEST

Resolved

The incident was fully resolved with the fixes done yesterday around 19.50h.

Posted Apr 03, 2024 - 10:04 CEST

Monitoring

We've identified the issue as being caused by a failure in our internal infrastructure, which was just remedied. The printing and downloading of labels should now work correctly again in all cases. We are monitoring the situation again, and will resolve this incident post tomorrow morning when higher customer volumes prove that the fix has worked.

Posted Apr 02, 2024 - 20:42 CEST

Identified

We're seeing that the issue still persists for the downloading of labels and documents from our portal. We will continue to solve this incident accordingly.

Posted Apr 02, 2024 - 16:53 CEST

Monitoring

The issue has been solved by restarting some API related services internally, we are still investigating the root cause and monitoring the issue closely.
Should you continue to experience the same printing issue still, please contact our support department.

Posted Apr 02, 2024 - 16:48 CEST

Investigating

We are currently investigating a printing issue in our platform. Several customers report that they cannot print labels through their WMS/ERP anymore, and are also stating that downloading labels from our my.transsmart.com portal does not work leading to time-outs and printing errors.
The current theory is that an underlying part of the label infrastructure is down causing all of the label issues but we hope to state that with more certainty here soon. We're investigating with the highest priority and will revert here when we know more.

Posted Apr 02, 2024 - 16:19 CEST

This incident affected: Infrastructure.