Our postmortem concluded that the problem was caused by a new database operation that had not been properly vetted for performance. We’ve implemented a better process for testing and diagnosing these problems going forward. Additionally, we are rescheduling our sprint deploys for the weekend so that future changes will not cause disruptions during business hours.
Posted Mar 04, 2019 - 15:16 PST
Resolved
Confirmed that performance is back to normal.
Posted Mar 04, 2019 - 14:21 PST
Monitoring
The problem has been located and fixed, after an extended search. All services back up. We are monitoring for any further problems.
Posted Mar 04, 2019 - 14:07 PST
Identified
This morning's deploy caused a performance problem with a background job. This appears to be causing some slowness across the application. We're turning off the offending job so we can diagnose the problem. Performance should return to normal shortly.