How Sandy Has Altered Data Center Disaster Planning
March 13th, 2013 By: Rich Miller
“We’ve had a lot of conversations with ConEd,” said Shobowale. “They are very aware of the issue. They’ve developed a capital plan to address this.”
Shobowale, who was appointed last year by Michael Bloomberg, emphasized that while few concrete solutions have been put in place, the city is working hard on a process to address the broader impacts of climate change, including flooding from more powerful storms. that includes the location of mission-critical equipment, and the type of equipment being used.
“Let’s get this equipment out of the basements,” said Robert Bianco, VP and General Manager of TW Telecom. “We’re putting this equipment on higher levels of buildings.”
Sabey Data Centers, which is about to open its Intergate Manhattan facility, is housing its diesel fuel tanks in the basement, but is anchoring them and equipping them with submersible pumps, said John Sasser, VP of Operations. The company’s building was not affected by flooding during Sandy, and is well above even the revised flood plains, he said. But the company is taking no chances.
Fuel pump failures in basement-level diesel tanks were key failure points for several buildings that were flooded during Sandy, including 33 Whitehall and 75 Broad, where staff of several providers formed a “bucket brigade” to relay 5-gallon buckets of diesel fuel up 17 flights of stairs to refuel a generator providing emergency power to a Peer 1 data center.
So why are these tanks in the basement? New York imposed restrictions on rooftop diesel storage after the Sept. 11 terrorist attacks, when an early engineering analysis from FEMA suggested that leaking fuel from diesel tanks contributed to the collapse of 7 World Trade Center. “We had recently applied to move the fuel tanks to the roof and were rejected,” Alex Reppen, the CEO and founder of Datagram, told DCK in November. “We’re hoping the city may change its view on fuel. We’ve talked to a lot of our competitors, and believe that this issue will be revisited.”
Shobowale said yesterday that New York’s policies on the location of fuel tanks are being discussed, but said the city must also balance safety considerations associated with storing diesel on higher floors in buildings.
Looking Ahead: An Altered Readiness Landscape
New York’s disaster readiness challenge is being examined on many levels, from the details of codes to fundamental questions about shoreline redevelopment and the best ways to protect the city’s infrastructure, businesses and residents from the Sandys to come.
“We are a city of islands,” said Shobowale. “We have more coastland than Los Angeles, San Francisco and Boston combined. We have a dense, urban environment which is much more difficult to (adapt) than just putting a building on stilts in Louisiana.”
If you’re weighing “lessons learned” and want to implement some of them in your data center or disaster recovery operation, you’d better get your budget requests submitted soon, panelists said, as the storm’s grip on C-suite attention may have a shorter shelf life than you think.
“A year from now, Sandy will be a more distant event,” said Hines. “Memories fade. You need to strike while the iron is hot. This is the time to do it. A year from now, it will be much harder.”
A better solution, get your servers and DR infrastructure out of Manhattan !!
Kazuhiko YPosted March 16th, 2013
Natural disaster would be beyond all imagination. So, the place by itself is a secondary matter, which means every place could suffer from natural disasters.
In my opinion, the best solution is to establish procedure to stop the facilities of data center in unimaginable natural disaster as being able to restart data center in a short term, while all people understand natural disaster has a tendency to exceed all imagination. Of course, it comes near to stating the obvious that dispersion of risk is important.
P BulteelPosted March 19th, 2013
Fukushima should have been an eye opener 2 years ago that any critical system in a basement is going to be subject to potential flooding. At that point everyone should have evaluated where these systems sit and prepared by not housing those critical systems in a basement or having secondary backup systems that are not in the basement. Hindsight is 20/20 – but 2 years ago we saw it with Japan. Shouldn’t something have been learnt from that?