We've determined stability has been reestablished.
Posted Dec 23, 2021 - 20:54 PST
Monitoring
The rollback appears to have fully restored functionality. We are continuing to monitor the situation closely to ensure stability.
Posted Dec 23, 2021 - 19:26 PST
Update
We're preparing to roll back to yesterday's version of the application as a temporary fix so we can investigate this problem more deeply without extending this outage further.
Posted Dec 23, 2021 - 19:18 PST
Update
The previous fix was ineffectual and we are working on another. We're certain now that it is a networking issue.
Posted Dec 23, 2021 - 18:53 PST
Update
We've developed and are in the process of publishing a new patch.
Posted Dec 23, 2021 - 18:34 PST
Identified
The previous solution seems to have only worked temoporarily; we're continuing to develop a permanent fix.
Posted Dec 23, 2021 - 18:25 PST
Monitoring
Our API servers appear to be responding normally once again. We will continue monitoring to ensure stability.
Posted Dec 23, 2021 - 18:18 PST
Identified
We've identified the issue is related to a recent update and are reloading our servers now.
Posted Dec 23, 2021 - 18:10 PST
Update
We've localized the issue to our API servers and are continuing to investigate.
Posted Dec 23, 2021 - 17:48 PST
Investigating
We're aware of an issue causing certain internal and external API calls to time out or take unusually long and are currently investigating.