Timeline
A scheduled restart of nrt-03 began, as part of the weekly restart cycle in the Chiba data center.
nrt-03 did not come back. Health checks withdrew it from new routes, and within a minute clients were sending new messages through nrt-01, nrt-02, nrt-04 and kix-01.
Messages already waiting in mailboxes held on nrt-03 could not be collected. Some recipients in East Asia began to see delays.
The on-call engineer in Chiba found the relay stalled at boot, waiting for a storage controller that had not initialized.
After a cold start, nrt-03 came back. Waiting messages were collected and delivered, and delivery times returned to normal.
What happened
For 47 minutes, some people in East Asia received messages late. About one in thirty of the region’s recipients had a mailbox on the affected relay; for them, messages waited for up to 47 minutes. Everyone else saw no change, because new messages were routed around the relay automatically.
No messages were lost, and none could have been read. Messages wait in mailboxes encrypted end to end, and the relay holding them cannot open them, whether it is working or not.
The cause was a storage controller that failed to initialize after the restart. We replaced it two days later, and have since replaced the same model in all four Chiba relays. Restarts in Chiba are now staggered, and each relay hands its mailboxes to a neighbor before it restarts, so a relay that does not come back no longer leaves anyone waiting.