After Sandy: Datagram Recovers From ‘Apocalyptic’ Flood
November 26th, 2012 By: Rich Miller
“It was desolation,” said Reppen. “The streets were like a riverbed, with mud and garbage everywhere. I saw a filing cabinet and a typewriter lying in the street. It was 36 hours before we could walk out to the lobby without being up to our waist in water.
“At first, pumping was futile,” as there was water everywhere and no place for it to go. “Once the sewers started working, we were pumping like crazy. It really took 2 to 3 days solid days of pumping (to empty the basements).”
With the fuel pumps badly damaged, Reppen and his team began searching for portable street-level diesel backup generators that could restore power to the data center.
“We ordered six generators, and only one arrived,” said Reppen, who said one generator was sold out from under them when the generator owner received a better offer while the unit was en route. Other delivery attempts resulted in repeated delays from heavy traffic and restrictions on access to the flood zone in lower Manhattan. It wasn’t until the afternoon of Friday, Nov. 2 that the 2-megawatt Caterpillar generator arrived outside 33 Whitehall. By the following day, services were restored.
A Street Full of Generators
“Once we got the physical generator on site, we were on our way,” said Reppen, who said two other tenants, Verizon and Cogent, also have mobile generators on-site.”We have a whole street full of generators.”
The generators also allowed power to be restored to the building’s elevators – which was key for Datagram, whose staff had been using the stairs to access the company’s data center on the 25th floor. Once the backup generator was on-site, the repairs continued on switchgear, and the basement pumps. Once those were fixed, the rooftop diesel fuel “day tank” needed to be cleaned, polished and refilled.
With those systems now back online, Datagram continues to operate on generator power. ”We should have at least temporary feed from ConEd (this) week,” said Reppen. The building is still not open to most tenants.
Datagram was founded in 1994 as an ISP and managed services provider. After a period of growth in colocation facilities, in 2004 the company opened its data center at 33 Whitehall, a 30-story building also known as the Broad Financial Center. The building was the original NASDAQ headquarters, and a major telco hub for PSINet and Verizon Business.
aPosted November 26th, 2012
I’m disappointed that the first priority wasn’t safety! A tech was sprayed with fuel in the very next paragraph. I’d like to know more about how they made sure everyone was safe in the midst of the chaos.
AngusPosted November 26th, 2012
@a: You are disappointed that safety isn’t a data center’s first priority? That doesn’t make much sense.
If safety was the “first priority”, they would close the data center immediately and permanently, because no one can get hurt if no one’s there. But like everyone else, they make a reasonable compromise between safety and productivity instead.
You are disappointed with them for not following an unrealistic and impractical ideal.
DonPosted November 27th, 2012
I’m disappointed to see major industry providers blaming the city and blaming the weather.
Why weren’t the pumps submersible? Submersible diesel pumps are nothing new, and neither is the concept of a NYC basement flooding. Fuel tanks themselves can be submerged to a given depth (depending on how the tank is engineered) without any issues — as long as a vent pipe is run to high enough ground.
Why weren’t the generators on higher ground? I understand NYC’s wonderfully complex codes prohibit rooftop FUEL storage, but what’s wrong with putting the generators up top — or at least on a higher floor? In-building generators are nothing new, and neither are the compliance requirements of doing so. Pulling fuel up to the higher floor is going to require some additional cost, but that brings me to my point.
They didn’t do these things because they’re too expensive. It was cheaper to pile all your mission-critical gear in the basement. And in doing so, you got caught with your pants down.
We aren’t talking about Discount Don’s Datacenter-O-Rama here. We’re talking about Verizon. Cogent. Major players that are generally near the top end of the pricing spectrum. It’s not that they can’t afford to build a truly reliable datacenter, it’s that they wanted to save a few bucks by putting all their eggs in one basket.
I’m glad to see Datagram is heeding my advice and moving everything up a few stories, but why wasn’t this done the first time? Who looks at a datacenter plan and says “Well, all our mission-critical redundancy infrastructure is in the building’s weakest point, but I’m sure that will be fine”?
Verizon, Cogent, and Datagram, apparently.
Chris HewittPosted November 27th, 2012
@Angus. These are data centers for the financial services industry. Do you really think someone’s personal safety is worth the risk? Even a first responder’s first step in every protocol is “scene safe / personal protective equipment.” Any risk to a data center tech’s personal safety is never a “reasonable compromise.” Would you risk personal injury or death to save a server?