A code change containing an edge case defect was deployed which caused a memory failure in the application. This defect was only encountered as part of a functional use-case that was not previously covered in standard regression tests.
When an end user utilized the assessment function which contained the code defect, the application consumed all available memory and crashed the application.
Code deployment was rolled back, which resolved the issue.
09/11/19 7:11am US EDT - Application deployment executed which injected defect into SkillCheck application.
09/11/19 7:45am US EDT - First occurrence of defect encountered which took down the application. ST DevOps and Engineering teams engaged to begin investigation.
09/11/19 8:00am US EDT - Application brought back online after failure.
09/11/19 8:25am US EDT - Second occurrence of defect encountered which took down the application.
09/11/19 8:30am US EDT - Application brought back online after failure.
09/11/19 8:55am US EDT - Third occurrence of defect encountered which took down the application.
09/11/19 9:00am US EDT - Application brought back online after failure.
09/11/19 9:05am US EDT - Root cause identified and application deployment rolled back. Services fully restored to normal operation.
SkillCheck
Additional test case added to deployment testing scripts to target specific functionality / use case identified with defect.
Additional regression test automation scripts being developed to widen test coverage for this and further edge case scenarios.