Outage with data processing in US East and UAE North
Resolved
Mar 27 at 08:20am AEDT
Service restored
Affected services
Created
Mar 26 at 05:00pm AEDT
This incident message has been published retrospectively
What happened:
Two of the credentials used in Data Studio (for US East and UAE North) expired
Our synthetic tests did not pick this up correctly and therefore it didn't trigger our on-call process
This meant users were unable to process data from around 0200 to 1715 26 March NYC Time in US East and 1000 26 March to 0100 27 March Dubai time
We have now successfully rotated the secret and restored service
Actions:
- Update synthetic tests to pick up this issue in the future
- Improve our process to identify upcoming secret expiries for this particular credential type
Affected services