What happened?

An issue was encountered with one of the deployment steps during the execution of a scheduled Career Website platform release. As an attempted remediation, a manual action was executed which was not part of the standard deployment process. This action had the unintended effect of dropping the configuration data of all plugins on the career website platform.

What was the impact?

During the incident, Career Websites were not available and did not display the correct branded site, they simply displayed a default “Enfold” placeholder page.

Resolution

The plugin configuration data had to be restored from the database backup in order to recover functionality of the Career Website platform.

Incident Timeline

04/19/2019 - 03:03am US EDT - Deployment failure occurred and monitoring alerted of site availability issues.

04/19/2019 - 04:02am US EDT - Config data restore initiated.

04/19/2019 - 06:45am US EDT - Career Site services restored.

What products / customers were impacted?

Career Websites

X-Cloud Employee Sites

Corrective Actions Undertaken to Prevent Recurrence

Employee reinforcement training on release protocol & exception authorization process.

Investigation and diagnosis underway on cause of original deployment failure which resulted in troublesome manual workaround.

Release process operational review to be conducted.

Posted Apr 19, 2019 - 15:50 EDT

Resolved

Platform performance has stabilized and sites are operating normally.

Incident post-mortem details will be posted to this incident within 24 hours.

Posted Apr 19, 2019 - 07:46 EDT

Monitoring

Sites have been recovered, but are currently running slowly as site contents are recaching and the platform is catching up on backlogged events.

Slowness is expected to continue for approximately 30 min to 1 hour while recovery completes.

The Symphony Talent team is continuing to monitor the recovery progress.

Posted Apr 19, 2019 - 06:56 EDT

Update

Recovery steps are progressing.

Data restore is complete and server configs are currently being redeployed to pick up updates.

Posted Apr 19, 2019 - 05:56 EDT

Identified

The source of the issue has been identified.

Metadata was dropped during a failed deployment step of a scheduled deployment. The Symphony Talent team is currently working to restore the metadata and bring services back online.

Posted Apr 19, 2019 - 05:12 EDT

Investigating

We have identified an issue which is impacting hosted career websites and X-Cloud Employee sites.

Sites are displaying a generic Enfold theme page rather than site contents.

The Symphony Talent team is investigating the issue and will provide more details as soon as possible.

Posted Apr 19, 2019 - 04:29 EDT

This incident affected: SFX - Career Websites (CWS) and SFX - CRM.