Repairs Force Seattle Data Center Offline for 5 Days
August 27th, 2012 By: Rich Miller
The city of Seattle will turn off many public services for five days over Labor Day weekend so it can repair the power system supporting its primary data center, Seattle Mayor Michael McGinn said Thursday. The shutdown will allow the city to replace a faulty bus bar providing power to the city data center on the 26th floor of the Seattle Metro Tower.
During the shutdown, which begins Friday and runs through Tuesday, city residents will be unable to conduct many services they would normally access online, including paying bills and applying for licenses. Critical operations, including 911 and other public safety systems, will remain online.
“The city will certainly be open for business, but it won’t be quite as convenient as it is normally,” said Seattle Administrative Services Director Fred Podesta. “But this is a short-term event, and it’s a holiday weekend.”
Workarounds for Faulty Bus
The repairs were prompted by overheating of the primary bus bar, which McGinn described as “a big beam that brings electricity from the street up to the data center.”
“The temperature of the bus had gone up because it wasn’t conducting electricity properly at a joint where two pieces come together,” said McGinn in a news conference. “The temperature was going up, and that’s not good. It could have led to a failure, and some sort of arcing or short. Upon learning this, we immediately began shutting down servers and moving systems and reducing the load on the bus, and that brought the temperature back down to a safe and stable temperature. But we knew we would need to replace the bus and take care of that bad joint.”
For a lesson in the hazards of a bus failure, McGinn won’t have to look far. In July 2009 an electrical fire at Seattle’s Fisher Plaza disrupted Internet commerce and caused $6.8 million in damages to the building, a major communications hub housing data centers. The fire was caused by a failure of two bus ducts – enclosures housing copper bars to conduct electricity – in a basement power room of Fisher Plaza. The incident caused extended downtime for the payment portal Authorize.net, which left thousands of merchants unable to process credit card payments through their web sites.
Crippled in Calgary
Last month the city of Calgary was hit by a data center fire that crippled city services and delayed hundreds of surgeries at local hospitals. The incident serves as a wake-up call for government agencies to ensure that the data centers that manage emergency services have recovery and failover systems that can survive as series of adversities.
In March the city launched an analysis of its data center operations, which will assess whether Seattle should build more data center space, use a third-party provider or move services to a public cloud.
“When we put the data center on the 26th floor (in 2000), we didn’t anticipate the explosion of data and the number of servers we would be using,” said McGinn. “We need greater redundancy in that system. We need to upgrade our center. This is not the most reliable way for us to support the city’s operations.”
Erin Devoto, the Deputy Director of Information Technology for Seattle, says a study of the city’s data center options is underway and will take about a year.
I was reading this and the post about the datacenter fire. I can’t imagine what would happen if we turned off the power for 5 days ? Out of business ?
Does anyone know how large the fire was ?
kind of ironic that a (the?) banner ad at the bottom of the article is for Internap – one of the main providers in Fisher plaza
JeffPosted August 27th, 2012
This points to the differences in the relative value of downtime across the datacenter world. Some orgs just don’t feel it that bad when the outage is managed properly (like in this case, scheduling it for a low traffic period.) Orgs that require higher availability even in circumstances like this should already be running a remote DR site with the ability to take over all workload in the event of the main site needing to go completely down.