Career Site Job Search Issue

What happened?

An issue with the job search API serving jobs to some career sites caused job searches to display no results.

What was the impact?

During the incident, some career sites were unable to display job search results until the issue was resolved.

Resolution

An SSL certificate scheduled for rotation was only partially implemented across the necessary hardware points. Due to the incomplete implementation of the new certificate, the job search API was impacted when the old certificate expired. An additional contributing factor was the misconfiguration of operational monitoring of the job search API, as it was only being monitored over http. Once identified, the SSL certificate was updated and operational monitoring was udpated to monitor the API via https.

Incident Timeline

09/19/17 4:37am US EST - Jobs API issue began.

09/19/17 6:41am US EST - Issue identified.

09/19/17 7:06am US EST - Issue resolved.

What products / customers were impacted?

Some Career Websites

Corrective Actions Undertaken to Prevent Recurrence

SSL Certificate documentation updated to reflect missing target instance so it's accounted for in future certificate rotations.

Operational monitoring updated to check http and https for the target.

Additional monitoring setup to further monitor health of Jobs API.

SSL certificate rotation procedures being reviewed to ensure full coverage / accuracy.