Datacenter outage at Equinix DC6 last Tuesday burned me hard

I was sitting at my desk in Seattle when our primary link to Equinix DC6 went dark for 3 hours. Turned out their BGP session dropped because of a misconfigured route filter on their end. Now I make sure every single provider gives us a written SLA on prefix limits and filter changes. Has anyone else dealt with carrier failures that cascaded because of a simple config mistake?

2 comments

2 Comments

jade2265d ago

Yeah because nothing says "fun Tuesday" like watching your entire production pipeline fold because some dude fat-fingered a route map. I bet their change ticket just said "improving stability" too.

white.grant4d ago

Bet they blamed it on "human error" in the postmortem like it wasn't a time bomb their own process set up. You'd think after the fifth oopsie they'd start locking down those route tables, right?