Issue with messages missing in Storage & Playback, Push notifications in Europe PoP
Incident Report for PubNub
Postmortem

At 01:08 UTC on August 3rd we observed that messages published to persistence from our Europe PoP from 23:32 August 2nd to 00:41 UTC August 3rd were not accessible by History API call, and push notifications sent during the same period were not delivered. We implemented a fix and the issue was resolved at 02:04 UTC on August 3rd resulting in stored messages being backfilled and accessible as expected. Push notifications sent during the period in question remained undelivered due to expiry. This issue occurred because there was an error in our server code that went undetected, resulting in messages waiting in the queue unprocessed.

To prevent a similar issue from occurring in the future we expanded testing coverage to ensure messages are written to persistence as intended. In the coming days we will implement enhanced internal alerting to provide early warning of such an occurrence in the future.

Posted Aug 03, 2022 - 21:23 UTC

Resolved
This incident has been resolved.
Posted Aug 03, 2022 - 02:04 UTC
Monitoring
A fix has been implemented. Messages published to storage during the time period in question to our Europe PoP are now available for retrieval.

We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you'd like to discuss the impact on your service.
Posted Aug 03, 2022 - 02:00 UTC
Update
Implementation of the fix remains underway. We will provide an update within 30 minutes.
Posted Aug 03, 2022 - 01:43 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Aug 03, 2022 - 01:12 UTC
Investigating
Users who published messages to our European PoP from ~23:32-00:41 GMT may be temporarily unable to access them. No errors are being returned from the service, despite messages not being returned correctly. Push messages sent during that time were not sent to their intended recipients. The other services, including PubSub, are working normally. While the issue has been resolved, we are working now to restore access to the stored messages.
Posted Aug 03, 2022 - 01:08 UTC
This incident affected: Realtime Network (Storage and Playback Service, Mobile Push Gateway) and Points of Presence (European Points of Presence).