SkillCheck issue identified
Incident Report for Symphony Talent
Postmortem

What happened?

A code change containing an edge case defect was deployed which caused a memory failure in the application. This defect was only encountered as part of a functional use-case that was not previously covered in standard regression tests.

What was the impact?

When an end user utilized the assessment function which contained the code defect, the application consumed all available memory and crashed the application.

Resolution

Code deployment was rolled back, which resolved the issue.

Incident Timeline

09/11/19 7:11am US EDT - Application deployment executed which injected defect into SkillCheck application.

09/11/19 7:45am US EDT - First occurrence of defect encountered which took down the application. ST DevOps and Engineering teams engaged to begin investigation.

09/11/19 8:00am US EDT - Application brought back online after failure.

09/11/19 8:25am US EDT - Second occurrence of defect encountered which took down the application.

09/11/19 8:30am US EDT - Application brought back online after failure.

09/11/19 8:55am US EDT - Third occurrence of defect encountered which took down the application.

09/11/19 9:00am US EDT - Application brought back online after failure.

09/11/19 9:05am US EDT - Root cause identified and application deployment rolled back. Services fully restored to normal operation.

What products / customers were impacted?

SkillCheck

Corrective Actions Undertaken to Prevent Recurrence

Additional test case added to deployment testing scripts to target specific functionality / use case identified with defect.

Additional regression test automation scripts being developed to widen test coverage for this and further edge case scenarios.

Posted Oct 07, 2019 - 17:52 EDT

Resolved
The ST team has verified that SkillCheck continues to operate normally.
Posted Sep 11, 2019 - 14:05 EDT
Monitoring
A fix has been implemented and service has been restored. The team is currently monitoring and investigating the root cause.
Posted Sep 11, 2019 - 08:41 EDT
Investigating
We have identified an issue which is impacting the availability of the SkillCheck online job assessments functionality.

The Symphony Talent team is investigating the issue and will provide more details soon.
Posted Sep 11, 2019 - 07:59 EDT
This incident affected: SkillCheck - Assessment Testing.