QUEsocial Issue Identified

Incident Report for Symphony Talent

Postmortem

What happened?

A domain controller failure resulted in the QUESocial and X-Cloud Employee applications becoming unavailable and inaccessible.

What was the impact?

During the incident time frame, users were unable to log in to the application.

Resolution

Domain controller was repaired and brought back online, restoring application functionality

Incident Timeline

01/10/2019 - 05:23am US EDT - Initial failures detected.

01/10/2019 - 06:43am US EDT - Root cause identified and domain controller rebuild process initiated.

01/10/2019 - 01:45pm US EDT - Services fully restored

01/10/2019 - 11:30pm US EDT - Permanent fix implemented to remove affected routing from domain controller to prevent future recurrence.

What products were impacted?

QUESocial

X-Cloud Employee

Corrective Actions Undertaken to Prevent Recurrence

Immediate Recovery - domain controller rebuilt and brought back online.

Long Term Fix - Affected routing removed from domain controller and implemented via more stable mechanism for prevent future recurrence.

Posted Feb 08, 2019 - 13:41 EST

Resolved

The issues affecting QUEsocial and the X-Cloud Employee app have been resolved.

The Symphony Talent Operations team is continuing to monitor for any further issues.

Posted Jan 10, 2019 - 15:08 EST

Update

The Symphony Talent team is continuing to troubleshoot the backend connectivity issues affecting the application and will update again soon.

Posted Jan 10, 2019 - 12:26 EST

Investigating

XCE and QUEsocial are experiencing a temporary outage due to an issue with our AWS servers. Our Development Operations team has identified the issue and is working to rectify it as soon as possible.

Users will be able to access the login screen but will not be able to log in. They will also be unable to share content during this time. Similarly, admins will be able to access the login screen for the admin portal but will be unable to log in.

We will update again by 10 am CST.

Posted Jan 10, 2019 - 09:37 EST