What happened?

A database index issue prevented the QUEsocial & XCloud Employee applications from being able to properly make updates in the database.

What was the impact?

During the impacted timeframe, users observed dramatic performance degradation or the application became completely unresponsive.

Resolution

An index maintenance routine was found to be missing which should have properly cleaned up the index on a nightly basis. Further review identified that this maintenance routine had not been migrated over when a one-time infrastructure enhancement project was completed two months ago. The index maintenance process was put back in place and the index was rebuilt.

Incident Timeline

05/31/2019 - 11:25am US EDT - Alerting and monitoring identified issues with the QUEsocial and XCloud Employee applications.

05/31/2019 - 12:02pm US EDT - Indexing issues identified as root cause.

05/31/2019 - 01:45pm US EDT - Healthy index restored and services functioning normally.

What products / customers were impacted?

QUEsocial

X-Cloud Employee

Corrective Actions Undertaken to Prevent Recurrence

Index maintenance routines were put back in place to conduct regular ongoing maintenance.