Slicehost

This is a precautionary notification that the local utility company will be working on the community service feed for the STL-B data-center building from July 23 0400 GMT – July 24 0100 GMT). During this window, we will be off utility power and running on generator power. There will be multiple representatives on hand from the utility company, the building operations group, our data-center and Slicehost. We do not expect any customer impact during this window but wanted to relay the notification we received. As always, please let us know if you have any questions.

UPDATE:

No issues thus far, but they have extended the maintenance window an additional 7 hours with plans to resume usage of the community power feed at July 24 0800 GMT.

UPDATE:

All done. Smooth as silk.

The blog has been a little quiet as of late, but we've been busy working on several updates.

CentOS Updated

CentOS was recently updated to version 5.2, now available for new Slices and rebuilds. (If you want to upgrade an existing CentOS installation you'll have to do so using the standard CentOS upgrade path.)

Newer Kernels

We have added support for newer 2.6.24 derived kernels. Currently all new slices are built with these kernels, if you'd like to update any existing slice, let us know via a support request.

SliceManager Security Updates

We've been hard at work on the SliceManager, fixing bugs and adding features. Community feedback has been an invaluable resource, notably in a recent discussion in the SliceForum regarding SliceManager security. We have several options on the table, most of which are still under discussion, but we decided to push out a couple of simple ones right away:

• Notify me on SliceManager login failures
• Send Slice root password via email

These can be found under Accounts > Security Preferences. Additionally, we now only allow up to 3 login failures in 30 minutes, significantly reducing the possibility of a brute force break-in to your SliceManager account.

Several updates and features are on the horizon, so stay tuned. If you have any feedback, feel free to leave a comment, forum post, email, or drop by our chat.

As of Friday, January 25, the Slice Manager has the ability to display simple diagnostics for your Slices. It will display whether your Slice and/or host server are up, the load on the host server, and the rate of swap usage (reads and writes per second). Additionally, we’ve added links to allow you to use the Just-Ping service to ping your Slice, or your own internet connection, from over 20 points around the world.

Datacenter 1 will have a network maintenance window Friday Dec. 14 between 0601-1200 GMT. At this time, BGP configurations will be modified and BGP sessions moved to the new core (which we were physically moved to last week). The expected impact is momentary outages affecting some customers as BGP updates propagate, with most of these changes taking place between 0700-1000 GMT. We’ll be available in the chatroom during the window. As always please contact us with any questions.

Today we’ve enabled a few notifications for Slicehosters. These are sent to the customer email addresses, not the billing email addresses.

Swap

The first notification is about swap usage. We’re monitoring pure I/O on each Slice’s swap partition, and if it exceeds our current threshold (subject to change) you’ll receive an email. This will hopefully let you know that the slice is using more swap than it should. This is bad because using excessive swap will degrade performance dramatically. You have a 3-day threshold to rectify the situation before you are emailed again.

We’ll follow up next week with some ways to tackle swap usage on your slice.

Blocklist

This monitor watches the IPs we provide on 2 Spam Blocklists with more to come soon. The email you receive gives a link to the Blocklist website. There you will find information on how to remove your IP(s) from that list.

Bandwidth Overage

You shall receive this notification the day you go over your bandwidth allotment. There is no change in service, but you will be charged an extra $0.30 (USD) per extra gigabyte of transfer.

Datacenter 1 is in the final stages of moving us to a new core network and we’ll be moved this Friday Dec 7 between 0700 and 0900 GMT. There will be a network outage of 5-10 minutes during this window as we are physically moved onto the new network. We’ll be in the chatrooms during the window. As always we apologize for this downtime, but ultimately it should result in a more stable network.

Network maintenance will be performed on Friday Nov 16 at our first datacenter from 0600-1100 GMT. Expected impact will be momentary network outages as traffic is migrated to the new core from 0700-1000 GMT. This should be one of the last major maintenance events as we move onto the new core at this facility. We’ll be online during the window. As always, please contact us with any questions.

No nightclubbing for us this Saturday night, instead we completed an emergency maintenance – upgrading firmware in all of our switches to circumvent a bug that started popping up. You shouldn’t have noticed any downtime, this was done between 0500 and 0700 GMT at both datacenters.

We’ve been running on our backup connection most of the day. Around 1645 GMT yesterday we failed over and remained there while the NOC researched what happened. At 0800 GMT they will be moving us back over, people may see brief outages as the routes are updated.

Your prayers to the NOC gods have been answered! Earlier this week we turned up our second facility and started burning through the waitlist once again. What does this mean?

Current customers You won’t notice a difference, SliceManager handles all of this transparently. When you add a slice, it’s kept on the same local network as your other slices for fast communication. If you want to add slices at another facility for redundancy, the best way to currently do this is creating another account. That may change in the future, but we wanted to keep things simple to start. If you’d like to have slices in both facilities, send us an email and we’ll push you through the waitlist.

New customers It’s been a long and painful journey, but things will get better fast. We started tearing through the backlog this week and should get everyone onboard faster than the estimates on the waitlist page. Pretty soon, you’ll be able to tell all the newbies to wait patiently in line!

Future We’re pretty pumped because this is the fruit of several months of labor. These changes allow us to grow easily in both the short and long term. And we can get cracking on some swell stuff we’ve been dreaming up for the past several months.

We just received word of a datacenter maintenance event scheduled this Friday. It will affect most customers, although not those who signed up this week. During the 4-hour window, the service impacting activities are scheduled to occur between 0800-1000GMT.

• routing config changes on core routers
• software updates on core routers
• replacing interface card with a defective port

The expected impact is 3-4 outages of 5 minutes or less. We’ll be monitoring during the window, remember to check status.slicehost.com, slicechat and our twitterstream for updates.

Update

These are the time frames to expect network outages if everything goes as planned:

0800 to 0810 08330 to 0835 0900 to 0910

FYI:

• Ubuntu Gutsy (7.10) is available
• Gentoo is now 2007.0
• CentOS is now 4.5 (5 is coming soon)
• Bug fixes for all images

The datacenter network maintenance that caused the outage yesterday was rolled back and will be performed again tonite (Nov 3 0800 GMT). It was supposed to be non-impacting, as is tonite’s maintenance. The Cisco bug that caused the problem has a work around and it will be employed this evening.

We’ll be monitoring things on our end – should there be problems remember to check the chatrooms, Twitter or status.slicehost.com. This maintenance is in preparation for the core cutover scheduled for next week.

Relaying more information as we receive it: the cause of the outage was due to a Cisco IOS bug dealing with HSRP. Unfortunately, it should have been circumvented and can be attributed to human error during the maintenance procedure. We’re meeting with the datacenter leads next week to discuss the issues of the past week and what is being done to correct them. There’s also discussion of moving the core cutover up, keep an eye on the blog/forum for any news on that front – we’ll send an email if it is happening sooner than expected. Again we are truly sorry for an unacceptable round of outages. As always, please contact us if you have any questions or concerns.

For those of you just tuning in, there was another network outage from 0900-1100GMT. We’re still piecing together what exactly happened but here is what we know so far:

• there was a non-impacting maintenance schedule by our datacenter in preparation for the core upgrade taking place in about a week. They were inserting a switch in parallel with the existing core switches.
• starting at 0900 GMT when the maintenance was completed, the outage started.
• after working with Cisco, the issue was resolved around 1100 GMT

This is an unacceptable level of service and people have expressed concerns in the chatroom about the recent outages. We agree completely and cannot offer more than our sincerest apologies at the moment. We are gathering more information and reviewing our options, updates will follow. In the meantime, if you have any questions or feel like venting, you can contact us via email or call me directly (314.266.3502). If I don’t answer, leave a message and one of us will get back to you shortly.