The Objects service is reporting errors and high latency in the EU PoP
Incident Report for PubNub
Postmortem

Problem Description, Impact, and Resolution

At about 17:33 UTC, the Objects service began to report errors and experience elevated latencies in the EU PoP. We started to implement rate limits on the impacting Objects query, during that process the latency resolved itself by 19:10 UTC. The reason for the issue is due to a lack of monitoring to detect runaway query processes within the Objects service.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring we have implemented the necessary monitoring to alert our on-call staff and established default rate limits to prevent incorrect queries from consuming resources unnecessarily.

Posted Dec 01, 2021 - 15:43 UTC

Resolved
The incident has not resurfaced since 19:10 UTC. We are resolving this issue, and we will follow up with a post-mortem soon.

We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service.
Posted Nov 24, 2021 - 19:47 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 24, 2021 - 19:19 UTC
Identified
At about 17:33 UTC, the Objects service began to report errors and experience elevated latencies in the EU PoP. PubNub Technical Staff is investigating and more information will be posted as it becomes available.

If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).
Posted Nov 24, 2021 - 19:16 UTC
This incident affected: Points of Presence (European Points of Presence) and Realtime Network (Objects Service).