Explosion at The Planet Causes Major Outage
June 1st, 2008 By: Rich Miller
An electrical explosion and fire Saturday at a Houston data center operated by The Planet has taken the entire facility offline. The explosion at 5 pm Saturday has affected 9,000 customer servers, and the company says it hopes to restore service by late Sunday afternoon. No servers or network equipment were damaged by the explosion, but the data center is without power. The Planet said it is working with the fire department and its facilities staff to restore power and get servers back online.
UPDATE: As of early Wednesday, power has been restored to the first floor (Phase 1) of the Houston data center. A generator failure early Tuesday slowed the recovery process at The Planet, leaving 1,500 servers offline and prompting the company to offer to physically migrate customer servers to a second Houston data center it operates. See our previous coverage of the damage to the facility and industry feedback on potential lessons learned from the incident.
“(Saturday) evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room,” said Doug Erwin, CEO of The Planet, in a message on the company’s forums (mirror). “Thankfully, no one was injured. In addition, no customer servers were damaged or lost.”
Early indications are that the fire was caused by a short in a high-volume wire conduit. The fire department is not allowing the company to run backup generators, so the facility has been without power since the incident occurred. In the latest update, The Planet says the damage to the data center was more extensive than initially believed.
The explosion affected only the main Houston data center, with no impact on any of The Planet’s other five data centers. The company hosts more than 50,000 servers and 22,000 customers in its six data centers, meaning that about a third of its customers and 20 percent of customer servers are currently offline.
This was the second time an explosion and fire has occurred at the Houston data center, which had a transformer explode in June, 2003 when the company was known as Rackshack.
Many of the customers affected are accounts from EV1Servers, the Houston-based dedicated hosting specialist that was acquired by The Planet in 2006 as part of a larger transaction in which private equity firm GI Partners bought both companies. The Houston data center houses the servers for ServerCommand, the management portal for former EV1 customers. “We are in the process of moving the ServerCommand servers to other Houston data centers so that we’re able to loop them into communications,” the company said.
The Planet has sought to move accounts to get customers back online, with limited success. “During the early stages of the H1 data center we opportunistically relocated some customers to another data center,” wrote Urvish Vashi, the Director of Product Management for The Planet. “However, due to network and data center (power/cooling) constraints, this option is no longer available and requests for migration cannot be honored.”
The Planet’s main page was knocked offline briefly, according to monitoring from Netcraft, but was back online in less than an hour. The Planet’s forums are also online, but are experiencing serious availability problems due to traffic, including a Slashdotting.
“This is a significant outage, impacting approximately 9,000 servers and 7,500 customers,” said Erwin. “All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.” Center Networks has noted the strong customer communication during the outage.
The Planet said it intends to proactively credit accounts for downtime under terms of its service level agreements (SLAs). The Plant also operates data centers in Dallas, where it was headquartered until last year, when its main offices were shifted to a new facility in Houston.
I have been bitting my nails for some 32 hours now and I am absolutely fumming over this.
Read the following and and see if you can’t connect the dots.
My conclusion is this guy gets and award while my clients and I get stuck sucking wind.
Both have photos and information directly provided by ThePlanet staff
This is crazy. definately one of the biggest outages I have heard of.
Just relax, enjoy the quiet time. Maybe go fishing.
soon to be former customerPosted June 2nd, 2008
This has been handled very poorly. I don’t understand the “praise” for communication.
1. The first notification went out 6hrs after the incident.
2. The update messages didn’t provide any real info – just “we’re continuing to work on restoring power.”
I found out about the plan to get the 2nd floor up and running on another site hours before the Planet communicated that. There were messages that “more important” customers were having their servers moved to the second floor to be brought back up sooner, as well as other message boards of “important customers” having their machines moved to other locations. Aren’t all customers important?
3. The tech staff was virtually useless – precanned responses.
When I first learned of the issue, I asked about switching machines or changing DNS and the tech support person said I was taking a gamble as they’d be up by noon Sunday. I switched DNS as I didn’t want my sites down any longer.
The next day when I asked again whether it was better to just get a new machine, they agreed, but the best part is their offer was for a more expensive machine with less features than the one I just got 2 weeks ago. They wouldn’t even match the specs of the machine or the price – what kind of customer service is that?
Whoever out there thinks The Planet is handling this great is crazy.
At this point as it closes in on 2 days downtime with really no vote of confidence in being fully restored, they probably could have packed all the servers up and moved them to other locations during the same time period.
I too will dump these guys as soon as I get all the data off.
Anyone with recommendations on server solutions should post them to these forums.
I am planing to remove my nails as I feel it will damage my head as I am scraching my head from last 48 hours due to shutdown of my website
NinaPosted June 3rd, 2008
The way that the company has handled this has been completely unprofessional! We are into our third day without access to our account and it has had a significant impact on our business. What is even worse, no one from the company has contacted us or even responded to our e-mail and phone calls!
I bet that when it comes time to renew your accounts, the company will once again send you the “we have a fantastic service record” e-mail! Absolutely awful!!
Sadly, this isn’t the first time The Planet has had issues in their facilities, and I’m sure it won’t be the last. From what I’ve heard from customers calling up to switch service, and from what others have posted here in the comments, it appears that they aren’t handling it particularly well either.
For those customers who are upset with the lack of service and communication they are receiving from The Planet, I would like to offer my company as an alternative solution. I represent GNi (Global Netoptex, Inc.). We are the largest supplier of IBM Blade servers in the USA, and we have multiple data centers available to suit the needs of any client, large or small. We are currently having a summer sale on blade servers in our San Francisco, CA blade center, as well as colocation in our Chandler, AZ, Los Angeles, CA, and Oakland, CA facilities.
Our NOC services are world-renowned, and our Chandler, LA and Oakland facilities have never experienced a severe outage such as this. We have techs available 24/7/365, and they are always available. When you call our facilities as a customer, you will always get a live, on-site technician at the other end of the line, whether it’s at 3 PM or 3 AM. I think it’s ridiculous that some customers on here have not received a personal response from the company within three days.
If you would like a different datacenter experience – one where quality of service and performance is of the utmost importance – please give GNi a call, or send me an e-mail. I’ll take care of you personally.
(408) 850-8202, or e-mail firstname.lastname@example.org. Best of luck making it through this outage, and I hope that whoever you choose as your next data center gives you a much better experience. And don’t worry if you’re located in Texas – our dedicated servers require no hardware installation on your part (or if you do wish to use colocation, you can ship your servers directly to our facility and we’ll rack them for you!)
TracyPosted June 3rd, 2008
I did extensive research following this debacle, and I purchased a hosting package with inmotionhosting.com. They’re highly rated with the BBB, and their staff has been extremely professional and courteous in our dealings. I just need these guys to get back up so I can get my data.
Frustrated Seanic clientPosted June 3rd, 2008
If this company has any class whatsoever, they should offer compensation for everyone’s lost revenues and clients. $1000 a day would be a good place to start.
It’s inexcusable that they didn’t have some sort of back up system to deal with this type of event, and then led us along hour-by-hour to keep us from leaving. Downright criminal.
ChrisPosted June 4th, 2008
I think some of you are missing a point. 9000 customers were affected – do you really think they have time to reply to all those emails and phone calls? There was only so much they could do in this situation. In my opinion (as an affected customer who lost quite a bit of revenue in the downtime) the updates posted on their forums were sufficient. My biggest gripe was their backup generator was housed so closely to the main power. Seems like an obvious design flaw.
While it’s a good start for them to post those updates on the forums, I do think they should be taking the time to contact every customer who calls them or sends them an e-mail personally. We strive to always have someone on the other end of the line available and to never let a customer reach an answering machine, as well as 15 minute response time by e-mail. While I understand that things are going to be slower in a crisis situation like this, having no personal response from the company when one was requested within three days is, in my opinion, unacceptable.
stonerPosted June 8th, 2008
Yes all the bad service is true, but to me, the reason why I am gone is when they said go to the forums for news and updates we did only to have one of their managers start calling customers “trolls” I was banned when I posted telling him how sad it was that at a time of crisis he chose to call the very people who send in money every month that paid his check trolls. Then to make matter worse not ONE of his bosses have responded to any of my emails. I guess the planet really do think the customers are trolls other wise they would have sent out an apology.
I just went now to get this link and now a different moderator say if folks keep asking for details about what happen he will start closing threads. I have NEVER seen a company try to hide facts like this before and I have NEVER send money to be called names… I am out of there.
I told someone yesterday that when everything settles down, what will be remembered is not that they had an outage, equipment breaks down, we all know that. But how in a time of crisis, they called their customers trolls.