Increased Latency and Errors in Presence in the US (East and West) and AP-North regions

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

On July 8, 2025 at 17:15 UTC on, we observed elevated errors and increased latency for customers using our Presence service in the US East, US West, and Tokyo regions. We identified an abnormal concentration of traffic and applied a configuration change to redistribute load across our infrastructure. The issue was resolved by 17:45 UTC the same day.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring in the future, we have implemented targeted balancing for Presence traffic patterns that exhibit such a concentrated load. This allows us to distribute traffic more evenly across infrastructure components. In the coming days, we will work to identify additional patterns that may require similar configuration changes to ensure even load balancing.

Posted Jul 09, 2025 - 00:48 UTC

Resolved

With no further issues observed for the past 30 minutes, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced an impact related to this incident, please report it to PubNub Support at support@pubnub.com.
Posted Jul 08, 2025 - 18:16 UTC

Monitoring

Services have returned to normal and we will continue monitoring for the next 30 minutes.
Posted Jul 08, 2025 - 17:46 UTC

Update

We are continuing to investigate this issue.
Posted Jul 08, 2025 - 17:39 UTC

Investigating

At approximately 17:18 UTC, PubNub Presence services began experiencing elevated latencies and server errors in the Asia Pacific-North and US regions. PubNub Technical Staff is currently investigating, and more updates will follow once available.
Posted Jul 08, 2025 - 17:36 UTC
This incident affected: Points of Presence (North America Points of Presence, Asia Pacific Points of Presence) and Realtime Network (Presence Service).