Storage service is experiencing elevated latencies in the US-East PoP

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

At 08:57 UTC (12:57 PST) on 2021-11-25, we observed elevated latencies and errors for the Storage service in the US-East PoP. The elevated latency caused the inability to retrieve messages from storage within the expected timeframe.

This issue occurred because our storage vendor was running deployments in the US-East PoP region which resulted in higher storage write latencies. We contacted our storage vendor to postpone the deployments to later in the day when traffic levels were lower. The issue was resolved at 14:08 UTC.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring in the future we will coordinate optimal times for off-peak hours deployments.

Posted Dec 01, 2021 - 15:49 UTC

Resolved

We are in communication with our Storage vendor who is performing rolling deployments to update their infrastructure. They have put a hold on US-East updates until later during non-peak hours.

Deployments to the AP-Northeast (Tokyo) PoP are scheduled to begin at 16:00 UTC (08:00 PST). The deployment is estimated to run for 3 to 4 hours. There could be similar latencies in that PoP for Storage during that time period.

We are resolving this incident. If there is any further impact to this (US-East) PoP or others, we will post a new incident at that time.

We apologize for the impact this may have had on your service. Please feel free to report any issues or questions about this incident to the PubNub Support Team (support@pubnub.com).

Posted Nov 25, 2021 - 14:08 UTC

Monitoring

We have seen the latency return to normal and has been operating as expected for the past 45 minutes (since 11:49 UTC or 03:49 PST). We will continue to monitor this closely for at least the next 30 minutes before we determine that this incident is resolved.

If you continue to experience issues that you believe are related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Nov 25, 2021 - 12:37 UTC

Identified

At 08:57 UTC (12:57 PST) on 2021-11-25, we observed elevated latencies and errors for the Storage service in the US-East PoP. This is directly related to a maintenance deployment being performed by our storage vendor. The estimation from the storage vendor is that this will continue for about another 90 minutes.

PubNub Technical Staff is investigating and more information will be posted as it becomes available.

If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Nov 25, 2021 - 11:45 UTC

This incident affected: Points of Presence (North America Points of Presence) and Realtime Network (Storage and Playback Service).