Slicehost

Technical Nitty Gritty

As scheduled, there was backbone maintenance at our NOC overnight, all went as planned (minor network flaps during the window). Just before the outage, a core router (upstream from us) began to have trouble and ultimately stopped routing. Normally this wouldn’t be a problem, everything would fail over to a secondary core router. However the BGP sessions stayed up to both the failover and Verizon from the primary – so the secondary took no action. Bouncing the BGP sessions had no effect. The fix required removing some routes and restarting the device (earlier bounce attempts had failed).

Post Mortem

We cannot apologize enough for this outage – it is simply unacceptable. Jason and I want Slicehost to be an infrastructure for our customers to build upon. Downtime does not factor into that vision. We’re working on changes which we hope will simplify networking, give us more control and improve reliability at the same time. An offsite server to handle communication during an outage will be in place as soon as possible. It was certainly frustrating for those outside of the chatrooms to go for an hour without updates. We’ll let everyone know how to reach this site once it is online.

Slicehost has experienced unbelievable growth and built an amazing community since we launched last year. Our customers have high standards. They demand technical efficiency, transparency and communication. We are devoted to serving this community and meeting its changing needs. Thanks so much for choosing our services, we know you have many options. Please let us know if you have any questions.

PS – We really appreciate the words of encouragement several folks sent our way, having been in similar situations themselves.