So what are the lessons learned from Datagram’s experience with Sandy?
“The biggest lesson learned is redundancy,” said Reppen. “A large number of customers buy services that are redundant. A lot of what we do is to guarantee 100% uptime, and many of our customers had a seamless experience. But I think a lot of complacency has built up over the years, and it left a lot of people at risk. A lot of customers opted out (of redundant hosting).
“In the future we’re going to speak up with our customers that are single-homed,” he added. “That’s ‘pre-Sandy thinking.’ We’re not seeing this a sales opportunity, but we intend to be very direct in helping them understand the consequences.”
While he’s not without a stake in the issue, Reppen doesn’t believe the flooding and downtime is likely to diminish the demand for data center space in lower Manhattan, as some have suggested.
Customers ‘Need to be Downtown’
“We have customers that are downtown with us because they need to be downtown,” said Reppen. “They have no option. If you need a lot of connectivity, the tip of New York is the best place for that. Jersey or uptown are not the answer for these customers. There are other customers that really didn’t need to be downtown, and most have already moved to Connecticut or one of our other facilities.
“We haven’t lost any customers,” he added. “We have one or two that are unhappy with us and likely to move, but our larger customers have all been solid.”
Reppen also said the outage provided an interesting test of the “cloudability” of complex infrastructures. “Some customers who are single-homed tried to go to Amazon and other clouds, and ran into a lot of trouble adapting their environments,” he said. “Once they figured it out, their bills were 2 to 4 times what they were paying us. I think a lot of people discovered very quickly that thinking the cloud is going to save you is ludicrous.”
Moving to the Mezzanine
One of the most common questions about the outages at data center providers in Lower Manhattan concerns the location of the diesel tanks and fuel pumps. Why would you put mission-critical equipment in a basement in a potential flood zone? According to Reppen, the answer is simple: because the city wouldn’t let them put it on the roof.
New York’s restrictions on rooftop diesel storage tanks arose from the Sept. 11 terrorist attacks, when an early engineering analysis suggested that leaking fuel from diesel tanks contributed to the collapse of 7 World Trade Center. The building wasn’t struck by a plane, as was the case with the Twin Towers, but was damaged by debris from the collapse of 1 World Trade Center. The official report from FEMA gave credence to this theory, but a subsequent in-depth technical analysis from the NIST found that the diesel fuel was not a significant contributor to the fires and subsequent collapse. But the restrictions persist.
“We had recently applied to move the fuel tanks to the roof and were rejected,” said Reppen. “We’re hoping the city may change its view on fuel. We’ve talked to a lot of our competitors, and believe that this issue will be revisited.”
Despite the ban on rooftop fuel storage, Reppen said that Datagram and the building management at 33 Whitehall realize that housing critical infrastructure in the basement is no longer viable. A project is underway to move the diesel fuel storage tanks, pumps and switchgear to the mezzanine level. Located just above a three-story atrium at 33 Whitehall, the mezzanine level is about 35 to 40 feet above street level – above flood risk, but low enough that the city will allow fuel storage. The project is expected to be completed sometime in the first quarter of 2013.
“We’re prepared to make the investment,” said Reppen. “It’s investing in our customers and our future. Being downtown is critical for us. There’s not a whole lot we can do about the location except fix it.”
Datagram is also addressing the difficulties in procuring a mobile generator. “We’re going to buy a street generator and park it in our Connecticut data center,” said Reppen. “Then we just need a driver to get it to New York.”
The lesson of Sandy is that in envisioning risks to uptime, you need to think beyond your previous experience. “We’re still thinking about what the next disaster could be,” said Reppen. “We’ve had a terrorist attack. We’ve had a flood. Maybe a plague of locusts jamming the air intakes.
“This has been a a tremendous learning experience, which we’re not going to waste.”