10:27AM CST - We initiated a new code release that included an update to one of our dependencies, pylibmc. The build failed to complete the rollout to a portion of the storefront servers which resulted in intermittent storefront failures. A rough initial impact investigation tells us that this affected roughly 1200 distinct server request and about 600 people.
10:56AM CST - The code release was rolled back to a previous version which reduced the number of intermittent failures but left some users in a broken state due to a cookie being set on their browser.
11:18AM CST - We approved a fix and released to one server in order to monitor and verify the fix before full-scale release.
11:27AM CST - The fix was rolled out to all production servers.
11:30AM CST - Fix is live, we are continuing to monitor the issue.
Posted Feb 16, 2017 - 11:35 CST
Investigating
We are experiencing occasional storefront outages and are currently investigating the issue.