After Sandy: Datagram Recovers From ‘Apocalyptic’ Flood
November 26th, 2012 By: Rich Miller
NEW YORK - As SuperStorm Sandy came ashore on the evening of Monday, Oct.29, the staff at Datagram believed they were as ready as they could be, and hunkered down for a busy night. They had no idea how busy.
“We had our NOC (network operations center) on call, and we had been been testing generators to make sure they were ready,” said Alex Reppen, the CEO and founder of Datagram, a managd hosting provider. “We felt very confident that we would weather the storm.”
Mother Nature had other ideas. As the storm surge from Sandy pushed into the south end of Manhattan, water poured into the streets surrounding 33 Whitehall Street, home to Datagram’s primary data center.
“It was apocalyptic,” said Reppen. “It was like a tidal wave over lower Manhattan. Cars were picked up and swept away. We began to see these incredibly powerful surges of water into our basements. It was absolute chaos.”
Water quickly filled the building’s two basement levels, which house the diesel fuel tanks and pumps supporting Datagram’s emergency backup generators, as well as key switch gear. By then, Con Edison had already shut down the local power grid. As several of its best known customer sites went dark, Datagram began a week-long struggle to bring its storm-ravaged infrastructure back online.
On The Front Line of the Storm Surge
Sandy was a huge challenge for the entire New York/New Jersey data center industry, but the superstorm’s greatest impact was felt by a handful of facilities in the “Zone A” flood zone in lower Manhattan, whose basements and lobbies were flooded by the brutal storm surge. These buildings – which included 33 Whitehall, 60 Broad Street, 121 Varick Street and several Verizon facilities- confronted unprecedented damage as the storm came ashore.
A flood that damages mission-critical equipment is among the worst scenarios a data center can face, offering little hope for a quick fix. The extent of the problems facing 33 Whitehall became apparent quickly in the Datagram NOC.
“It was a lot happening, all at once,” said Reppen. “The water set off alarms on our building management system. It was like a Christmas tree. We systematically sorted through the alarms. We practice this, and we have procedures, but it was a lot of scrambling.”
The first task was to assess whether the diesel pumps in the basement remained operational, and could continue to support Datagram’s rooftop generator. The news wasn’t good.
‘Devastation Outside Our Windows’
“Our main priority was to keep our generator running,” said Reppen. “One of our technicians got showered with diesel fuel removing a solenoid (a valve to improve the flow of fuel through the supply line). That helped for a little while. But we saw the devastation outside our windows, and began to concentrate on cutting customers over (to backup facilities).”
Datagram owns and operates two data centers. In addition to the 16,000 square foot facility on the 25th floor of 33 Whitehall, the company also has a facility in Bethel, Connecticut, as well as colocation space at major New York and New Jersey data hubs. Many of Datagram’s customers, especially those in financial services, are “double-homed” and can operate their infrastructure from either location. The Datagram staff focused on helping those customers maintain their operations.
The news was less promising for customers with single-homed servers at 33 Whitehall, who were facing days of downtime. Water filled both basement levels and the building’s ornate lobby. The neighborhood was underwater.
aPosted November 26th, 2012
I’m disappointed that the first priority wasn’t safety! A tech was sprayed with fuel in the very next paragraph. I’d like to know more about how they made sure everyone was safe in the midst of the chaos.
AngusPosted November 26th, 2012
@a: You are disappointed that safety isn’t a data center’s first priority? That doesn’t make much sense.
If safety was the “first priority”, they would close the data center immediately and permanently, because no one can get hurt if no one’s there. But like everyone else, they make a reasonable compromise between safety and productivity instead.
You are disappointed with them for not following an unrealistic and impractical ideal.
DonPosted November 27th, 2012
I’m disappointed to see major industry providers blaming the city and blaming the weather.
Why weren’t the pumps submersible? Submersible diesel pumps are nothing new, and neither is the concept of a NYC basement flooding. Fuel tanks themselves can be submerged to a given depth (depending on how the tank is engineered) without any issues — as long as a vent pipe is run to high enough ground.
Why weren’t the generators on higher ground? I understand NYC’s wonderfully complex codes prohibit rooftop FUEL storage, but what’s wrong with putting the generators up top — or at least on a higher floor? In-building generators are nothing new, and neither are the compliance requirements of doing so. Pulling fuel up to the higher floor is going to require some additional cost, but that brings me to my point.
They didn’t do these things because they’re too expensive. It was cheaper to pile all your mission-critical gear in the basement. And in doing so, you got caught with your pants down.
We aren’t talking about Discount Don’s Datacenter-O-Rama here. We’re talking about Verizon. Cogent. Major players that are generally near the top end of the pricing spectrum. It’s not that they can’t afford to build a truly reliable datacenter, it’s that they wanted to save a few bucks by putting all their eggs in one basket.
I’m glad to see Datagram is heeding my advice and moving everything up a few stories, but why wasn’t this done the first time? Who looks at a datacenter plan and says “Well, all our mission-critical redundancy infrastructure is in the building’s weakest point, but I’m sure that will be fine”?
Verizon, Cogent, and Datagram, apparently.
Chris HewittPosted November 27th, 2012
@Angus. These are data centers for the financial services industry. Do you really think someone’s personal safety is worth the risk? Even a first responder’s first step in every protocol is “scene safe / personal protective equipment.” Any risk to a data center tech’s personal safety is never a “reasonable compromise.” Would you risk personal injury or death to save a server?