The City of Seattle’s 108 Degree Data Center
November 24th, 2008 By: Rich Miller
The temperature rarely reaches 100 degrees in Seattle. But the weather report inside the City of Seattle’s data center showed an unhappy 108 degrees on Nov. 16, after the the facility lost its cooling system. Bill Schrier, the CTO and Director of the city’s IT department, writes about the event on his blog.
The root cause of the outage was a failed power breaker on a pump for the water supply to the building housing the data center, which left the CRAC (computer room air conditioning) units without water. “We shut down a lot of servers and many services starting almost immediately,” writes Schrier. “Nevertheless the temperature in the data center rose to that toasty 108 degrees, setting a new record high (sort of) for Seattle.”
The data center supports about 500 servers, and is housed in “downtown 60 story skyscraper” (um, wild guess: Seattle Municipal Tower). Schrier also describes the city’s response in trying to keep critical services online while recovering the cooling system:
A dozen IT professionals showed up on site within two hours (despite interference from the traffic around a nearby Seahawks football game). The computer center manager – a 44 year employee and true hero Ken Skraban – was on site and immediately in charge. Two employees set up an IT operations center with an incident commander and support staff. Several responded to the data center and shut down servers in an orderly, pre-planned, color coded (red-green-orange-yellow) fashion, with the most critical servers (for example “Blackberry” support) staying up continuously. Server administrators from every major department in City government responded on site.
Schrier says the city is installing a closed-loop dry cooler system in January. “Disasters happen,” he writes. “Careful planning and skilled, trained staff will always mitigate their effects.”
WardPosted November 24th, 2008
Did they open a window?
William CollierPosted November 25th, 2008
We all know about the need for at least N+1 redundancy, maybe if we knew how much this ended up costing it might reenforce the argument to management. How many servers had to or have to be replaced?
RyanPosted December 9th, 2008
So, let me get this straight. They had a single pump connected to a single power source? What is the point of a datacenter if that is how it is configured? I’ve got some room in my basement with the same amount of redundancy. I’ll give them a great deal on the space.