Elevated rate of 5XX errors Subscribe US West Point of Presence (POP)
Incident Report for PubNub
Postmortem

Problem Description, Impact, and Resolution 

At 18:36 UTC on August 9, 2022, we observed a small number of errors served across all PubNub services in our US-West PoP. We identified a configuration sync issue with a load balancer, removed it from service, and the issue was resolved at 19:29 UTC. This issue occurred because alerting did not include the specific configuration problem affecting the load balancer which had to be identified through investigation.

Mitigation Steps and Recommended Future Preventative Measures 

To prevent a similar issue from occurring in the future we are working to enhance server-level monitoring and alerting in the coming days. This will ensure we are immediately aware of any recurrence and can act quickly to resolve it.

Posted Aug 12, 2022 - 16:06 UTC

Resolved
This incident has been resolved.

We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you'd like to discuss the impact on your service.
Posted Aug 09, 2022 - 20:08 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 09, 2022 - 19:33 UTC
Identified
Beginning on 18:36 UTC 08/09, some customers may be experiencing a higher rate of timeouts and errors. We have applied a rollback and investigating this issue.
Posted Aug 09, 2022 - 18:56 UTC
This incident affected: Realtime Network (Publish/Subscribe Service) and Points of Presence (North America Points of Presence).