X-Cloud Analytics - Issue Identified
Incident Report for Symphony Talent
Postmortem

What happened?

An elasticsearch node exceeded all available resources and failed. This resulted in a cluster failure, which resulted in the unavailability and eventual loss of data indexes used to present Analytics data in the X-Cloud platform.

What was the impact?

During the incident time frame, users saw either static data or no data in the Analytics tab within X-Cloud Recruiter.

Resolution

The affected elasticsearch cluster was brought back to a healthy state, and then indexes were rebuilt.

Incident Timeline

01/14/2019 - 05:14pm US EDT - Initial failures detected.

01/14/2019 - 08:33pm US EDT - Investigation and recovery efforts determined that the cluster would require a node rebuild and extension. Rebuild initiated

01/14/2019 - 11:50pm US EDT - Elasticsearch node recovered and resources increased. Index rebuilding initiated.

01/15/2019 - 09:17am US EDT - Data index rebuilding completed for 98% of customers.

01/15/2019 - 01:05pm US EDT - Data index rebuilding completed for remaining customers.

What products / customers were impacted?

X-Cloud Analytics function in X-Cloud Recruiter

Corrective Actions Undertaken to Prevent Recurrence

Immediate Recovery - Elasticsearch cluster rebuilt.

Long Term Fixes

1) Elasticsearch cluster resources dramatically increased.

2) Additional proactive monitoring added to elasticsearch cluster to preemptively identify failure signatures experienced.

Posted Feb 08, 2019 - 18:56 UTC

Resolved
The issues impacting Analytics features in X-Cloud has been resolved and systems are now functioning normally.
Posted Jan 15, 2019 - 19:18 UTC
Investigating
We have identified an issue which is impacting the Analytics functions within the X-Cloud platform. Some analytics functions may be impaired or showing no data.

The Symphony Talent team is investigating the issue and will provide more details as soon as possible.
Posted Jan 15, 2019 - 14:52 UTC
This incident affected: Analytics and X-Cloud.