Incident Overview: 

The Web Application User Interface (https://app.addigy.com) experienced extreme latency due to a spike in long-running queries, depleting resources and causing system slowness. This was noticeable when going to  our website and seeing the page take 1-2 minutes to load. No notable impact was shown across running device actions.

‌

Root Cause: 

The incident was triggered by long-running queries affecting multiple data sources across different systems. This caused resource contention, leading to performance degradation. The precise cause of the query spike is being further evaluated for additional improvements.

‌

Detection and Response:

• Alerts were triggered prompting the Operations and Engineering teams to immediately begin investigating.
• Remediation efforts included adding additional optimizations to data sources, optimizing resource performance, and improvements to offending queries.
• Combining all above mentioned remediation efforts, the system recovered and performance stabilized.

‌

Remediation Actions:

1. Implemented a process to include new remediation efforts into existing process action steps. 
2. Implemented a new process to prevent database resource contention.