NEW YORK - As SuperStorm Sandy came ashore on the evening of Monday, Oct.29, the staff at Datagram believed they were as ready as they could be, and hunkered down for a busy night. They had no idea how busy.
“We had our NOC (network operations center) on call, and we had been been testing generators to make sure they were ready,” said Alex Reppen, the CEO and founder of Datagram, a managd hosting provider. “We felt very confident that we would weather the storm.”
Mother Nature had other ideas. As the storm surge from Sandy pushed into the south end of Manhattan, water poured into the streets surrounding 33 Whitehall Street, home to Datagram’s primary data center.
“It was apocalyptic,” said Reppen. “It was like a tidal wave over lower Manhattan. Cars were picked up and swept away. We began to see these incredibly powerful surges of water into our basements. It was absolute chaos.”
Water quickly filled the building’s two basement levels, which house the diesel fuel tanks and pumps supporting Datagram’s emergency backup generators, as well as key switch gear. By then, Con Edison had already shut down the local power grid. As several of its best known customer sites went dark, Datagram began a week-long struggle to bring its storm-ravaged infrastructure back online.
On The Front Line of the Storm Surge
Sandy was a huge challenge for the entire New York/New Jersey data center industry, but the superstorm’s greatest impact was felt by a handful of facilities in the “Zone A” flood zone in lower Manhattan, whose basements and lobbies were flooded by the brutal storm surge. These buildings – which included 33 Whitehall, 60 Broad Street, 121 Varick Street and several Verizon facilities- confronted unprecedented damage as the storm came ashore.
A flood that damages mission-critical equipment is among the worst scenarios a data center can face, offering little hope for a quick fix. The extent of the problems facing 33 Whitehall became apparent quickly in the Datagram NOC.
“It was a lot happening, all at once,” said Reppen. “The water set off alarms on our building management system. It was like a Christmas tree. We systematically sorted through the alarms. We practice this, and we have procedures, but it was a lot of scrambling.”
The first task was to assess whether the diesel pumps in the basement remained operational, and could continue to support Datagram’s rooftop generator. The news wasn’t good.
‘Devastation Outside Our Windows’
“Our main priority was to keep our generator running,” said Reppen. “One of our technicians got showered with diesel fuel removing a solenoid (a valve to improve the flow of fuel through the supply line). That helped for a little while. But we saw the devastation outside our windows, and began to concentrate on cutting customers over (to backup facilities).”
Datagram owns and operates two data centers. In addition to the 16,000 square foot facility on the 25th floor of 33 Whitehall, the company also has a facility in Bethel, Connecticut, as well as colocation space at major New York and New Jersey data hubs. Many of Datagram’s customers, especially those in financial services, are “double-homed” and can operate their infrastructure from either location. The Datagram staff focused on helping those customers maintain their operations.
The news was less promising for customers with single-homed servers at 33 Whitehall, who were facing days of downtime. Water filled both basement levels and the building’s ornate lobby. The neighborhood was underwater.