After Sandy: Datagram Recovers From ‘Apocalyptic’ Flood
November 26th, 2012 By: Rich Miller
So what are the lessons learned from Datagram’s experience with Sandy?
“The biggest lesson learned is redundancy,” said Reppen. “A large number of customers buy services that are redundant. A lot of what we do is to guarantee 100% uptime, and many of our customers had a seamless experience. But I think a lot of complacency has built up over the years, and it left a lot of people at risk. A lot of customers opted out (of redundant hosting).
“In the future we’re going to speak up with our customers that are single-homed,” he added. “That’s ‘pre-Sandy thinking.’ We’re not seeing this a sales opportunity, but we intend to be very direct in helping them understand the consequences.”
While he’s not without a stake in the issue, Reppen doesn’t believe the flooding and downtime is likely to diminish the demand for data center space in lower Manhattan, as some have suggested.
Customers ‘Need to be Downtown’
“We have customers that are downtown with us because they need to be downtown,” said Reppen. “They have no option. If you need a lot of connectivity, the tip of New York is the best place for that. Jersey or uptown are not the answer for these customers. There are other customers that really didn’t need to be downtown, and most have already moved to Connecticut or one of our other facilities.
“We haven’t lost any customers,” he added. “We have one or two that are unhappy with us and likely to move, but our larger customers have all been solid.”
Reppen also said the outage provided an interesting test of the “cloudability” of complex infrastructures. “Some customers who are single-homed tried to go to Amazon and other clouds, and ran into a lot of trouble adapting their environments,” he said. “Once they figured it out, their bills were 2 to 4 times what they were paying us. I think a lot of people discovered very quickly that thinking the cloud is going to save you is ludicrous.”
Moving to the Mezzanine
One of the most common questions about the outages at data center providers in Lower Manhattan concerns the location of the diesel tanks and fuel pumps. Why would you put mission-critical equipment in a basement in a potential flood zone? According to Reppen, the answer is simple: because the city wouldn’t let them put it on the roof.
New York’s restrictions on rooftop diesel storage tanks arose from the Sept. 11 terrorist attacks, when an early engineering analysis suggested that leaking fuel from diesel tanks contributed to the collapse of 7 World Trade Center. The building wasn’t struck by a plane, as was the case with the Twin Towers, but was damaged by debris from the collapse of 1 World Trade Center. The official report from FEMA gave credence to this theory, but a subsequent in-depth technical analysis from the NIST found that the diesel fuel was not a significant contributor to the fires and subsequent collapse. But the restrictions persist.
“We had recently applied to move the fuel tanks to the roof and were rejected,” said Reppen. “We’re hoping the city may change its view on fuel. We’ve talked to a lot of our competitors, and believe that this issue will be revisited.”
Despite the ban on rooftop fuel storage, Reppen said that Datagram and the building management at 33 Whitehall realize that housing critical infrastructure in the basement is no longer viable. A project is underway to move the diesel fuel storage tanks, pumps and switchgear to the mezzanine level. Located just above a three-story atrium at 33 Whitehall, the mezzanine level is about 35 to 40 feet above street level – above flood risk, but low enough that the city will allow fuel storage. The project is expected to be completed sometime in the first quarter of 2013.
“We’re prepared to make the investment,” said Reppen. “It’s investing in our customers and our future. Being downtown is critical for us. There’s not a whole lot we can do about the location except fix it.”
Datagram is also addressing the difficulties in procuring a mobile generator. “We’re going to buy a street generator and park it in our Connecticut data center,” said Reppen. “Then we just need a driver to get it to New York.”
The lesson of Sandy is that in envisioning risks to uptime, you need to think beyond your previous experience. “We’re still thinking about what the next disaster could be,” said Reppen. “We’ve had a terrorist attack. We’ve had a flood. Maybe a plague of locusts jamming the air intakes.
“This has been a a tremendous learning experience, which we’re not going to waste.”
aPosted November 26th, 2012
I’m disappointed that the first priority wasn’t safety! A tech was sprayed with fuel in the very next paragraph. I’d like to know more about how they made sure everyone was safe in the midst of the chaos.
AngusPosted November 26th, 2012
@a: You are disappointed that safety isn’t a data center’s first priority? That doesn’t make much sense.
If safety was the “first priority”, they would close the data center immediately and permanently, because no one can get hurt if no one’s there. But like everyone else, they make a reasonable compromise between safety and productivity instead.
You are disappointed with them for not following an unrealistic and impractical ideal.
DonPosted November 27th, 2012
I’m disappointed to see major industry providers blaming the city and blaming the weather.
Why weren’t the pumps submersible? Submersible diesel pumps are nothing new, and neither is the concept of a NYC basement flooding. Fuel tanks themselves can be submerged to a given depth (depending on how the tank is engineered) without any issues — as long as a vent pipe is run to high enough ground.
Why weren’t the generators on higher ground? I understand NYC’s wonderfully complex codes prohibit rooftop FUEL storage, but what’s wrong with putting the generators up top — or at least on a higher floor? In-building generators are nothing new, and neither are the compliance requirements of doing so. Pulling fuel up to the higher floor is going to require some additional cost, but that brings me to my point.
They didn’t do these things because they’re too expensive. It was cheaper to pile all your mission-critical gear in the basement. And in doing so, you got caught with your pants down.
We aren’t talking about Discount Don’s Datacenter-O-Rama here. We’re talking about Verizon. Cogent. Major players that are generally near the top end of the pricing spectrum. It’s not that they can’t afford to build a truly reliable datacenter, it’s that they wanted to save a few bucks by putting all their eggs in one basket.
I’m glad to see Datagram is heeding my advice and moving everything up a few stories, but why wasn’t this done the first time? Who looks at a datacenter plan and says “Well, all our mission-critical redundancy infrastructure is in the building’s weakest point, but I’m sure that will be fine”?
Verizon, Cogent, and Datagram, apparently.
Chris HewittPosted November 27th, 2012
@Angus. These are data centers for the financial services industry. Do you really think someone’s personal safety is worth the risk? Even a first responder’s first step in every protocol is “scene safe / personal protective equipment.” Any risk to a data center tech’s personal safety is never a “reasonable compromise.” Would you risk personal injury or death to save a server?