On Thursday, December 17 at 00:00 UTC (December 16 at 7p.m. ET), the VIP Platform experienced a widespread service interruption that lasted approximately 13 minutes. Some services, like uploading images to the file server, were impacted for up to 33 minutes.
We take incidents like this very seriously and would like to outline what happened, as well as the steps we have taken to help prevent future occurrences.
What Happened
Connections to the VIP origin-servers and VIP Files Service were disrupted because of an expired certificate. This happened because, although a new certificate had been issued and deployed, it did not get updated across the entire VIP network. While our existing monitoring caught the expiry, it did not catch that a subset of the network had not been updated.
Impact
During the service interruption, 93% of all requests were successful, while 7% returned a 503 error. This means that VIP site visitors may have encountered a 503 error when attempting to access the front or back-end of their VIP applications.
Timeline
- On December 17th at 00:00 UTC the old certificate still in use on the hosts expired.
- At 00:11 UTC we received the first reports of interruptions to uploading files for back-end users.
- At 00:20 UTC we received the first report of uncached requests not loading affecting external website visitors.
- At 00:33 UTC the issue was fully resolved with the correct certificate loading across the VIP Platform.
Future Prevention
We have already implemented additional safeguards and process improvements designed to prevent similar issues from happening again. These include (but are not limited to):
- Extending alerts for expiring certificates to all VIP hosts using a TLS certificate, in order to ensure that certificate updates are propagated to the whole platform.
- Improving the existing monitoring for TLS certificates with fine-tuned checks that initially trigger 30 days before a certificate expires and repeat with an increased frequency when the expiration date approaches.
Questions?
If you have any questions related to this incident, please open a support ticket and we will be happy to assist.
Responses