X-Cloud Analytics - Issue Identified
Incident Report for Symphony Talent
Postmortem

What happened?

An elasticsearch node exceeded all available resources and failed. This resulted in a cluster failure, which resulted in the unavailability and eventual loss of data indexes used to present Analytics data in the X-Cloud platform.

What was the impact?

During the incident time frame, users saw either static data or no data in the Analytics tab within X-Cloud Recruiter.

Resolution

The affected elasticsearch cluster was brought back to a healthy state, and then indexes were rebuilt.

Incident Timeline

01/14/2019 - 05:14pm US EDT - Initial failures detected.

01/14/2019 - 08:33pm US EDT - Investigation and recovery efforts determined that the cluster would require a node rebuild and extension. Rebuild initiated

01/14/2019 - 11:50pm US EDT - Elasticsearch node recovered and resources increased. Index rebuilding initiated.

01/15/2019 - 09:17am US EDT - Data index rebuilding completed for 98% of customers.

01/15/2019 - 01:05pm US EDT - Data index rebuilding completed for remaining customers.

What products / customers were impacted?

X-Cloud Analytics function in X-Cloud Recruiter

Corrective Actions Undertaken to Prevent Recurrence

Immediate Recovery - Elasticsearch cluster rebuilt.

Long Term Fixes

1) Elasticsearch cluster resources dramatically increased.

2) Additional proactive monitoring added to elasticsearch cluster to preemptively identify failure signatures experienced.

Posted 2 months ago. Feb 08, 2019 - 18:56 UTC

Resolved
The issues impacting Analytics features in X-Cloud has been resolved and systems are now functioning normally.
Posted 3 months ago. Jan 15, 2019 - 19:18 UTC
Investigating
We have identified an issue which is impacting the Analytics functions within the X-Cloud platform. Some analytics functions may be impaired or showing no data.

The Symphony Talent team is investigating the issue and will provide more details as soon as possible.
Posted 3 months ago. Jan 15, 2019 - 14:52 UTC
This incident affected: Analytics and X-Cloud.