Cache status graph for www.bbc.co.uk which shows a significant spike of "error" for ~7 minutes

Notices where this attachment appears

1. Embed this notice
   Neil Craig (tdp_org@mastodon.social)'s status on Friday, 24-Nov-2023 12:20:45 JST Neil Craig

   Hands up if you caused a global outage today...
   Just me?
   Sorry!

   I was making a change to our "outside the UK" CDN config today for www.bbc.co.uk & www.bbc.com & the change included 2 bugs which pre-testing didn't spot:
   - A regex typo which caused 404s on www.bbc.co.uk
   - An incorrect TLS cert on the CDN origin which caused 503s on www.bbc.com

   These caused ~7 minutes of significant global outage.

   I spent most of the afternoon writing tests to catch this for next time.

   #DevOps #WebDev #CDN