Massive Flooding Damages Several NYC Data Centers
October 30th, 2012 By: Rich Miller
Flooding from Hurricane Sandy has hobbled two data center buildings in Lower Manhattan, taking out diesel fuel pumps used to refuel generators.A third building at 121 Varick is also reported to be without power. There were also reports of outages for some tenants at a major data hub at 111 8th Avenue, and many other New York area facilities were running on generator power amid widespread utility outages.
NOTE: For updates on recovery efforts on Wednesday, see our follow-up story, New York Data Centers Battle Back from Storm Damage.
Both Internap and Peer 1 are struggling to continue operations at 75 Broad Street after basement-level flooding disabled critical diesel fuel pumps, leaving the providers no way to refuel generators on mezzanine floors.
Customers of Datagram were knocked offline Monday evening as water flooded the basement of its building at 33 Whitehall, knocking out high-traffic sites including Gawker, Gizmodo, Buzzfeed and Mediate.
Both 75 Broad and 33 Whitehall were located in the “Zone A” flood zone, which was inundated by a storm surge of more than 13 feet as Sandy’s landfall in New Jersey coincided with high tide in lower Manhattan.
Meanwhile, Atlantic Metro reports that several of its data centers are offline, including its facility at 121 Varick, which has “significant flooding” in its generator room.
There were also multiple reports of downtime for some tenants at 111 8th Avenue, a major communications hub which is owned by Google. Some reports attributed the outages to network problems, while others said that an electrical panel had failed while parts of the building were switching to generator power.
Meanwhile, several data centers in northern New Jersey were reported to be running on generator power, as PSE&G and other utilities suffered widespread outages.
Datagram hosts a number of high-profile blogs, including the Gawker network, which remains offline Tuesday morning. “Gawker sites down after power cut off at Datagram, our data center down on Whitehall St,” tweeted Gawker publisher Nick Denton. “Backup power didn’t kick in fast enough.”
In fact, the problems at 33 Whitehall went beyond the speed of its power response. Other tenants reported that the basement had been flooded, knocking out diesel fuel pumps that support generators.
Buzzfeed said it was able to work with content delivery provider Akamai to restore a static version of its web site. It posted a short story about the problems experienced by Datagram, including a text sent to Buzzfeed from a Datagram official. “Basement flooded, fuel pump off line – we got people working on it now. 5 feet of water now,” the official wrote.
It wasn’t clear when Datagram would be able to restore service.
Diesel pumps were also the focus of problems at 75 Broad, where basement levels experienced major flooding that forced Internap and Peer 1 to shut down their operations at the facility.
“Around 9 PM the storm surge, combined with high tide, caused flooding to many portions of lower Manhattan, including in and around our data center at 75 Broad,” Internap said in a blog post. “At the peak there was three feet of flood water in the lobby at the site, causing both basement floors to be inaccessible. As a result of the flooding, both our redundant fuel pumps and our generator fuel tank were compromised and shut down. The system continued to run until all fuel within the secondary feeder tanks were exhausted and our facility lost power.”
The Internap facility lost power at about 11:45 a.m. this morning.
“We are working as quickly as possible to implement a workaround for the fuel system that will allow us to bring the generator farm back into operation,” Internap said. “It is unclear how long it will take ConEd to restore utility power to the site, but we are preparing for the possibility of remaining on generator power for many days.”
UPDATE: “We have coordinated fuel deliveries and pumps, as well as engineers to fabricate pipe to bring the fuel directly to the generators on the mezzanine level,” Internap reported this afternoon. “There is currently no ETA available for the completion of these activities, but we are working through all options to bring the UPS infrastructure back online as soon as possible. We will continue to work toward resolution and
provide status updates with any new information.”
After attempting to arrange temporary diesel fuel storage and pumps, Peer 1 Hosting said early Monday that it will “implement a controlled shutdown of NY Data Center at 10:45 ET.” But as of 12:30 p.m., the site was still online, and the company said it was awaiting a diesel delivery.
“A fuel delivery is en route to the facility but due to the various street closures and general road conditions caused by Hurricane Sandy, (and) the expected ETA of the fuel truck is approximately 1 hour, Peer 1 said on its support forum. “If the fuel delivery is not able to be made before 2:00PM EST, PEER 1 will need to initiate a suspension of all services at our affected facility beginning approximately at 2:15PM EST.”
Peer 1 has not provided any updates since the 2:15 deadline.
Atlantic Metro / 121 Varick
On Tuesday morning Atlantic Metro reported that it was experiencing “widespread network and facility power outages due to Sandy and its flooding.” The company was experiencing significant problems at multiple data centers:
LGA1 (325 Hudson Street) – “We are working with local building crew to clear the area of the fuel pumps needed for Diesel generator power. Replacement fuel pump estimated to be received (by 9:30 p.m. Tuesday). Work will commence to restore service at that time.” Atlantic Metro says repairs at LGA1 will address a connectivity problem at LGA6.
LGA4 (121 Varick Street) – “ There is significant flooding in the generator room of this IDC. We are working with FDNY to safely investigate the area. (The) site is suffering from a damaged ATS. Crews will be called in first thing tomorrow morning to begin repair efforts.”
111 8th Avenue
There were multiple reports on Twitter of outages for some tenants at 111 8th Avenue, one of Manhattan’s largest buildings and a key hub for Internet traffic. Companies affected included Equinix, Voxel/Internap and XO Communications.
“NY9 experienced a failed generator that impacted service to several customers,” Equinix said on it blog, referring to its facility at 111 8th Avenue. “We made repairs and service was returned this morning. The site remains on generator power.”
111 Eighth Avenue is among the world’s most wired buildings. It was originally built as the Port Authority Commerce Building in 1932, and was redeveloped for telecom use by Taconic in the late 90s.
25 Broadway (Telehouse)
Telehouse reported that its facility at 25 Broadway in lower Manhattan has remained online throughout Sandy and its aftermath, as have its data center in Chelsea and at the Staten Island Teleport. The colocation provider said it switched to generator power proactively ahead of Con Edison’s utility power shutdown. “Since that time, these facilities have remained operational without any critical system (power, cooling) outages,” Telehouse reported. ” These facilities will remain fully staffed and off-grid (independent power generation mode), with periodic refueling until utility power is restored and stable.
“During this period, both Manhattan sites (25 Broadway, Chelsea) were operationally unaffected by local flooding,” the company added. “Building engineers and equipment, at both Manhattan facilities, are engaged in mitigating this threat. The Teleport facility, on Staten Island, had no flood threat during this time.”
32 Avenue of the Americas (CoreSite)
CoreSite reported that its data centers all remained online throughout Sandy, although its data center at 32 Avenue of the Americas in New York switched over to generator power Monday evening. “That transition occurred smoothly according to established procedures,” CoreSite said in a statment. “CoreSite is checking regularly with Con Edison as to when normal power is expected to be restored, but remains confident in its ability to continue running on its emergency generators as long as necessary given that fuel deliveries have been secured and remain on schedule.”
“Our thoughts go out to the people and communities affected by the devastation of this storm,” said CoreSite President and CEO Tom Ray. “Amidst this much bigger picture, we are striving to ensure our customers have timely and accurate information as to the current state of site operations and understand how to reach us.”
Colocation and interconnection specialist Telx is the largest service provider at 60 Hudson Street, one of the the leading carrier hotels in Manhattan. Telx says 60 Hudson lost power Monday at 7:15 p.m. but is operating on generator power, and has enough fuel to operate through Wednesday.
Telx also has facilities at 111 8th Avenue and says it is operating fine on generator power.
We will continue to update as we get more news. You can email us at firstname.lastname@example.org with news and tips.
Manhattan IT Support is on call to help client’s recover and restore data.
Susan UttendorfskyPosted October 30th, 2012
Ever read “When Sysadmins Ruled the Earth”? It’s an end-of-world scenario with the internet struggling to stay alive:
[...] hosting from Internap, Peer 1 and Datagram, which are located in lower Manhattan were affected. Data Center Knowledge has the story: “Flooding from Hurricane Sandy has hobbled two data center buildings in Lower [...]
I used to work in New York City quite a few years ago now. That’s just incredible how the water flooded so many places. Obviously people never envisioned the possibility their fuel pumps becoming submerged.
I’m sure there’s going to be a lot of learning opportunities from this storm.
Patriot Clean Fuel has emergency teams on standy-by for emergency fuel related issues including full reconditioning, water sepearation and removal, filtration and fuel procurement. We have mission critical partners in this area of Manhattan.
Jon WPosted October 30th, 2012
Amazing effort at 75 Broad St is documented at http://fogcreekstatus.typepad.com/ and http://fogcreekstatus.typepad.com/fog_creek_network_status/page/2/ . . . running drums of diesel fuel up the 17 flights of stairs?! Unreal.
Peter McPhamPosted October 30th, 2012
I wonder if this explains why YouTube was in read-only mode last night?
Steve GPosted October 30th, 2012
Online gaming for Call of Duty was also knocked out..
All the very best to the New Yorkers and the clean up operation. Let’s hope some measures can be put in place to prevent as much damage in the future.
MarkPosted October 31st, 2012
It’s a shame that the web based platform that I have to use everyday for my job is down since Sunday, with no back up website, not an email to mention that they are working on the issue. After all they have suffered a blackout and it’s unacceptable that they didn’t think to this kind of scenario before.
prov.net has a newly commissioned 400KVA rack ready 5000 square ft data center available in Providence RI for any customers in need of a quick turn-up alternative site
DC Rule of Thumb #101: Not Building Data Centers of ground floors [check]
[...] several data centers were knocked offline due to flooding — most notably Datagram, which hosts Gawker, Gizmodo, Buzzfeed, The Huffington Post, and [...]
TechUSA provides deployment teams to the environmental, construction, power & energy, and Information Technology sectors. Our group has a specific focus on mission critical facilities and mobilizes teams specializing in the installation, operation, integration, maintenance and repair of a physical plant and it’s subsystems. We have teams ready for deployment to service any facility in need of support during these difficult times.
StewartPosted November 1st, 2012
I could not report to work today, because 2 Broadway is flooded.
StewartPosted November 1st, 2012
Does anyone know the status of 2 Broadway. The last I heard is that the workers there will be charged vacation days until the building reopens. I think this is outrageous since this situation is unprecedented.
Let me know how we can help. I have a company that delivers Diesel to generators, construction equipment and boilers (heat). Let me know how I can assist you, Thank you
We could Have a truck onsite…
Our company had some back end servers in 75 broad st. I found a colo center in Orangeburg ny about 15 min outside NYC. They were able to provide us with a cabinet, power and ip in less then 6 hrs. I strongly suggest frontline data services for those who need a place to get your servers back up.
This was a real wakeup call for many providers and data center operators. Some have said this couldn’t have been predicted and is unprecedented but certainly the number of providers going down due to power and network issues was higher than most expected.
There’s only so much that can be done according to building codes but many providers didn’t have working generators or only had a single one.
With that said, there’s only so much that can be done in a situation where you have a shared pool of fuel stored in the basement that’s flooded and you aren’t allowed to store it higher up.
We’re glad to report our networks were unaffected, however the closest facilities we have are in NJ.
Many companies, big and small realized they need to have a disaster recovery plan as clearly many don’t have one.
Having seen how the seawater surge knocked out Japan’s nuclear reactors I would have thought that the management of datacentres should have had contingency for water damage. IT seems not. Please can all data matters worldwide start thinking about weather related problems that may occur.
R TroyPosted November 12th, 2012
MAYBE now management will begin to take DR planning and testing a bit more seriously, plus pay a lot more attention to how IT systems (and power delivery) are designed and located.
[...] escape the effects of this major natural disaster. Rich Miller from DataCenterKnowledge wrote an excellent piece on how New York Data Centers have been dealing with Sandy. Miller’s story outlines what Verizon, 111 8th, Datagram and several other data centers have [...]
Time to think about relocating these facilities to higher and less expensive ground. I am ready to assist anyone loooking to re-locate to the lower Hudson Valley region.
DineshPosted November 24th, 2012
Diesel generators are the best power alternative for electricity.
[...] looking at the reports, it seems like one of the most common issues was an inability to pump fuel up to the generators [...]
So, is there an alternative to these locations? What would be the perfect location for a data center in order to maintain everything needed for the uptime?