-
Data Center Migration Goes Awry at ValueWeb
July 31st, 2007 : Rich MillerA data center migration at ValueWeb has gone badly, leaving thousands of sites offline, some for as long as three days. The migration involved moving several thousand dedicated servers from Affinity Internet’s Miami, Fla. hosting facility to a Hostway data center in Tampa, about 270 miles away. The servers were leased by customers of ValueWeb and Gate.com, two business units of Affinity, which was acquired by Hostway in April.
The move, which began Friday night, was scheduled to take 12 to 15 hours. But by late Sunday night many customers reported that their sites remained offline. A forum post by a Gate.com manager indicates that all servers had been connected and powered on in the new location by 2 pm Sunday, but acknowledged that servers remained offline. Customers posting in a discussion thread at Digg said they were still down early Tuesday morning, more than three days after the migration started.
Among the sites knocked offline by the migration was HTML Help, a popular reference site for web site developers. “HTMLHelp’s two dedicated servers were offline for over 75 hours,” site maintainer John Pozadzides reported. “The worst part was that the company failed to communicate to anyone during the longer than expected outage as they went completely silent - at times not answering customer service phone calls, and sending no status update e-mails.”
For What its worth, my server was moved, and up about 24 hours later, pretty good as far as I’m concerned.
However, I did a complete backup, and ran all of my critical apps off a godaddy virtual server, in case they didn’t come back soon.. or at all 
Wow, I’m somewhat blown away. We moved our datacenter from San Francisco to Seattle this year, with no significant downtime for our customers. While we’re only a couple hundred servers, it was still a $250k project, and required 6 months of planning. We moved our customers over by buying enough servers for about 40% of our existing customers.
Once the new datacenter was built-up, migration was performed in four batches. Our new servers were built in Seattle, and all of the customer data and configurations were synced onto large NAS servers, which were shipped up to Seattle. Once in Seattle, the data was sycned from the NAS servers to customer servers, then we spent 4 weeks per grouping doing daily, then hourly, then 15-minute data syncs over the wire, and did late-night cutovers.
Perhaps this is why I’ve always disdained the commodity hosting market so much.
This is a key example of why it makes sense to outsource critical migrations. The cost of data center migration specialists would have been significantly less than what they will pay to recover from this, not to mention the bad press and lost customer loyalty.
Nina Labonte
Posted August 2nd, 2007Well it’s Thursday morning…1:51 a.m. EDT to be exact and our dedicated server has been down since last Friday. I was finally told at 12:45 a.m. that our “server” was not on Hostway’s list of servers w/ hardware issues. Nice…only took 5 days to find this out. But the mystery question nobody their can answer is…why then is it not online?
The customer service has been horrific, they either don’t respond or on the rare occasion you do get a live body they tell you..”I can’t give you an update because we’re in Ft. Lauderdale and your server is now in Tampa but we know they’re working around the clock to fix the issues”. Who’s clock? We’re 60+ hours and counting into this fiasco!
The idiots that planned this move need to be banned permanently from any type of IT work. Who in their right mind tries to move 3,500 servers in 12 to 15 hours? One could easily do the math in their head and figure out what a disaster this would turn into!
Once we come online - we’re gone. I’ll be shocked if Hostway doesn’t see a class action suit from this mess because they should.
Anon
Posted August 3rd, 20075 days here and our server is still offline. No communication. No nothing. I’ve never heard of something so disasterous.
my server with them has gone offline again, my server reboot request aren’t answered, …
they’ve been bad for a long time, now they’ve gone beyond that
RESOURCE LINKS:
