Technorati Struggles During Migration Prep
The popular blog tracking service Technorati has been having some performance issues this week, prompting the company to address the issues on its blog. Technorati site architect Ian Kallen explains that the problems are related to a pending data center migration, which is planned for this weekend.
Technorati has been hosted at 365 Main’s San Francisco data center, which suffered a major outage in July that knocked Technorati and several other major Web 2.0 traffic hubs offline. While power was restored to most of the affected colocation rooms within an hour, some of the affected sites took much longer to return to full service.
“Long prior to that event, we were already making plans to vacate 365 Main and move the infrastructure hosted there to another facility,” Kallen writes. “That incident hardened our resolve to do so with all available haste. Since then, we’ve been busily migrating functionality to another data center. Most of the time, these migrations have been transparent but there have been a few episode where things didn’t go quite as planned.”
Kallen said the Technorati team has been working to correct the unspecified problems, but that a “particularly egregious instance” occurred Wednesday morning, disrupting the site’s availability. Kallen said there could be additional downtime as Technorati moves to a new data center this weekend:
Beginning at midnight on the morning of Friday, September 21 (PDT), some delicate network functionality will be migrated out of 365 Main. While we expect any disruption to service availability to be at most a few minutes, we’ve identified a 4 hour window ending at 4:00 a.m. (PDT) within which these changes will be completed. We wanted to let everyone know that this was going to occur and hope to minimize any inconvenience this may present.
Data center migrations can be complex undertakings, but are usually completed with minimal disruption to a site’s users. But not always. About a week after the 365 Main outage, ValueWeb moved its dedicated servers from its Miami data center to a Tampa facility operated by Hostway Corp., which acquired ValueWeb’s parent company. The migration went badly, leaving thousands of customers offline. The company later said that more than 500 servers experienced hardware failures during the move.
While Technorati has decided to move out of 365 Main, other sites affected by the incident have stayed. Six Apart, which runs the TypePad and LiveJournal blog hosting services, decided to double down and added space in 365 Main’s new Oakland data center.
What I find difficult to understand is why this move is being conducted during the end of quarter. Business-wise, this month is when the US Gov’t budgets are due to be spent, and when most contract companies are making their purchases.
Second item is if Technorati had any type of DR drill in place and practiced so as to anticipate such a move.
Ironically, I’d bet 365 Main will have excellent uptime for the next few years. Technorati would do better to spread itself out over many facilities instead of just moving to another one. Every datacenter has a fairly equal risk of having SOME downtime. Equipment failure, or human error… Murphy will catch us all out at some point.
As for migrations, we performed one in early 2005. Moved over 1000 servers from our old facility in Bothell WA, to a new one in Seattle. It went flawlessly. One server failed, but was replaced within hours. I’d never want to do it again, but I know that if you plan right, execute right, it can be done.