Intermittent Storefront failures
Incident Report for Cratejoy
Resolved
This incident has been resolved.
Posted Feb 16, 2017 - 11:56 CST
Monitoring
10:27AM CST - We initiated a new code release that included an update to one of our dependencies, pylibmc. The build failed to complete the rollout to a portion of the storefront servers which resulted in intermittent storefront failures. A rough initial impact investigation tells us that this affected roughly 1200 distinct server request and about 600 people.

10:56AM CST - The code release was rolled back to a previous version which reduced the number of intermittent failures but left some users in a broken state due to a cookie being set on their browser.

11:18AM CST - We approved a fix and released to one server in order to monitor and verify the fix before full-scale release.

11:27AM CST - The fix was rolled out to all production servers.

11:30AM CST - Fix is live, we are continuing to monitor the issue.
Posted Feb 16, 2017 - 11:35 CST
Investigating
We are experiencing occasional storefront outages and are currently investigating the issue.
Posted Feb 16, 2017 - 10:57 CST