October 26th, 2007

Network trouble past couple of days

As I’m sure most people noticed, we had 2 major bouts of network trouble – yesterday afternoon and one today. It appears they were distributed denial of service attacks directed at multiple customers. The amount of traffic was filling both of the datacenter’s incoming pipes. There aren’t any excuses for why this took 2 extended outages to figure out and we’ll be addressing that issue with our provider. All I can offer are our sincere apologies and a promise that this will get better.

During the outages a lot of questions came up regarding multiple carriers, routers, etc. So to answer a couple of those:

  • we have redundant transport via XO and Verizon, however this won’t help since traffic will end up filling both.
  • we also have redundant routers using VRRP to failover on our side as does the datacenter.

Again, we’re extremely disappointed with how this was handled, but here is some good news. In the next couple of weeks, the datacenter’s core network (not ours, but how we connect to them) will be upgraded. We’re assured this will result in a more stable setup for growth.

Secondly and this was the announcement we referred to earlier this week, we are putting the final touches on a second facility. This space uses separate carriers and we will be managing the network, but everything else stays the same. We hope this expansion allows us to accelerate our growth and offer a higher level of redundancy for customers who need it.

Thanks to everyone for their patience during the past 2 days. For future reference, aside from the chatrooms, the slicehost twitterstream and the network status page are worth bookmarking for quick updates.

Posted by matt Filed in notification

10 Comments

  1. March 17th, 2008 at 10:00 PM stubblechin

    Thanks for the updates! I know I’m confident you guys will learn from the experience and this will only lead to an even better Slicehost.

  2. March 17th, 2008 at 10:00 PM stubblechin

    Great communication, by the way. I love that I have multiple channels for keeping tabs on the problem: IRC, campfire, twitter, offsite.slicehost.net.

  3. March 17th, 2008 at 10:00 PM Zach Holman

    I was kind of dismayed when I saw my site was down, and even a bit more angry when I couldn’t see any updates since slicehost.com was down. It was more just being out of the loop, really, that was frustrating. I remembered the Twitter stream and was pleased to see that it was updated with the details you had on hand. Definitely felt much better at that point; Twitter works fantastic in these situations.

  4. March 17th, 2008 at 10:00 PM Jeff

    Nice work!

  5. March 17th, 2008 at 10:00 PM Orion

    Thanks for keeping us informed about issues. DDoS happens with everyone, but IRC and campfire with latest news is a major advantage.

  6. March 17th, 2008 at 10:00 PM Bogdan Gaza

    When will the 2nd facility be ready? Thanks.

  7. March 17th, 2008 at 10:00 PM Joe

    I think its high time for a waitlist update in the blog. I’ve now waited twice as long as was predicted when I signed up (and before you froze sending out invitations).

    Its obvious things are taking longer than you expected (because they are taking longer than you predicted) but it would be good to let people know whether new slices are being opened yet, or whether we’re all on hold until the new facility comes online, or what the deal is…

  8. March 17th, 2008 at 10:00 PM Tim

    I second what Joe said. I’d love an update on the waitlist times and whether they’re currently realistic.

    Don’t get me wrong, I think it’s great that you don’t oversubscribe your servers – that’s one big reason why we’re so keen on you. (SliceHost is my first choice by far.) But the codebase for our little Rails app is almost finished, and the “2 weeks” wait time hasn’t changed for a week.

  9. March 17th, 2008 at 10:00 PM Brendan

    The DdoS attack was bound to happen at some point – that you’re learning from it is a healthy sign.

    It isn’t so hot that the upstream provider wasn’t overly prepared for what is a well known (and documented) load problem.

    It is surprising that there are still carriers out there who don’t seem to learn from the mistakes of others.

    @Tim,

    I believe the wait list times aren’t accurate and are somewhat fluid. They are at very best a guide.

    My wait period has gone from 9 weeks to 11 weeks. So not a promising sign despite comments suggesting things are going to happen faster.

  10. March 17th, 2008 at 10:00 PM Jason

    Sit tight, we should have a wait list update within the next day or so.

Leave a Reply