Generator Delivery Boosts Hopes at Datagram
November 2nd, 2012 By: Rich Miller
For the staff and customers of hosting provider Datagram, the cavalry finally arrived shortly after 2 p.m. Friday, in the form of a 2-megawatt diesel generator on the back of an 18-wheeler. The huge engine, which slowly made its way through the storm-battered streets of lower Manhattan to a data center at 33 Whitehall, represents the best hope for fully restoring the web sites of Datagram customers, which were knocked off the Internet Monday evening when the record storm surge from SuperStorm Sandy flooded key equipment in the building’s basement.
The power loss at Datagram provided the earliest warning that New York’s Internet infrastructure was facing serious trouble. Datagram hosts high-traffic web sites including the Gawker network, whose sites went offline Monday evening and were down entirely for a day before setting up temporary, ad-free sites on the free blogging service Tumblr.
It will take some time for Datagram to get customer servers back up and online. But the arrival of the generator has finally started the process of restoring power. By 2:30 p.m., the generator was in place and 30 workers were working to get it operational.
“Electrical switching equipment has all been installed,” the company said. “Generator cabling has arrived and is being set up. We are fully staffed and on-site. When power is restored network will be restored and then a systematic power up of all systems. We have had a lot of time to review and plan for this and are very eager to begin.”
Delivering and installing a 2-megawatt diesel generator is never a simple task under the best of conditions. Delivering one through New York under current conditions is another matter. Datagram’s first attempt at having a generator delivered resulted in repeated delays from heavy traffic and restrictions on access to the flood zone in lower Manhattan.
The new generator was scheduled to arrive by 5 p.m. Thursday afternoon. But at 8 p.m. Thursday it was still 100 miles away, stuck in traffic on Route 95. By 8:45 a.m. this morning (Friday) it had made it into the city and gotten as far as 26th Street. But by 11:45 a.m. there was still no sign of the generator, and the frustration was showing in Datagram’s updates to its customers.
“We are still waiting for work to be completed and generator to arrive,” the company wrote. “We do not understand why our generator is not here. We have permits, but they are being checked and rechecked almost every step of the way, causing very long delays at every turn.”
The company decided to make arrangements for a second generator. Which turned out to have been a good thing. “We have found out the generator on 14th is not safe and a possible fire hazard, most likely why it is being stopped at every point,” Datagram wrote. “We are actually lucky it did not arrive.”
Datagram is one of many data center clients that have sought to procure portable generators. Zayo Group, which had a generator delivery to 111 8th Avenue scheduled for later afternoon yesterday, said the unit finally arrived at about midnight.
“Our generator was coming from outside the city, and due to limited access to Manhattan and the enforcement getting into the city there were delays and congestion getting onto the island,” Matthew Brouker, Zayo Group’s Director of Marketing.
Chris ConnerPosted November 3rd, 2012
Why would anyone take the risk and host their servers/content in a coastal data center again. If you’re not stock market sensitive, move it.
JRPosted November 4th, 2012
sunken cost my friend…they are too financially committed to that location
DineshPosted November 6th, 2012
Diesel generators are the essential equipments now a days. Maintaining these generators is also a big task.
BrianPosted November 6th, 2012
The placement of power equipment in a basement looks economical, early on, but, unless the data center is built at the top of a hill, all too often becomes useless and costly as flooding occurs. With the exception of the power company’s transformer vault, all of the power and cooling equipment in our coastal (Pacific) data center was on the second floor of the building.
R TroyPosted November 12th, 2012
Companies have long located critical systems in low rent areas like basements. Generators and fuel tanks as well. It wouldn’t be so bad in the case of fuel tanks if they were sealed along with their pumps and wiring.
Allen IIIPosted November 12th, 2012
It is becoming OBVIOUS that generators, back up cooling systems for nuclear reactors need to be UP HIGH for SAFETY. Fukishimi-Daichi reactor melt down happened because the backup systems were on the ground instead of on top of the containment tower. Data centers need to think about this too, water always wil flood basements and therefore a back up system should be placed up high to avoid these kinds of things. This is the lesson l;earned from this storm!