Sept. 11, 2001: One Data Center’s Story
September 11th, 2011 By: Rich Miller
Major data centers and carrier hotels are designed to remain online through nearly anything. But the massive destruction wrought by the Sept. 11, 2001 terrorist attacks tested mission-critical facilities in lower Manhattan.
On today’s 10th anniversary of the attacks, America pauses to remember the profound impact of the event on our nation and the world, honoring those who were lost, and grieving with those they left behind. Many of the stories of Sept. 11 are ones of everyday heroism in dealing with the human side of the tragedy as well as the focus required to mount rescue and recovery efforts.
The events of the day, which included a loss of utility power for a large part of NYC, had a ripple effect that spread across many industries and businesses, which confronted challenges in providing critical services, including telecommunications. This included the data centers in the heart of the financial district in lower Manhattan, and especially 25 Broadway, a Telehouse facility located less than a half-mile from the World Trade Center that is home to the New York International Internet eXchange (NYIIX).
Generators Became Crucial
25 Broadway was not damaged in the attacks, but struggled to operate in a landscape of buildings and infrastructure shattered by the attack. Telehouse experienced fuel shortages and generator problems due to overheating (possibly tied to air quality issues and the dust shrouding lower Manhattan) that left customers off-line for parts of Sept. 13, 14 and 15.
At the time I was covering the industry for CarrierHotels.com. Here’s some excerpts from the coverage in September 2001:
“It’s been stressful and hectic from day one, and from minute one,” said John Shields, vice president of operations and engineering for Telehouse America. “There’s been a lot of uncertainty and a lot of chaos.”
The Broadway Center building near the south tip of Manhattan lost grid power from Con Edison soon after the World Trade Center’s twin towers collapsed, and quickly switched over to generator power. At the time, the building had three days of fuel on-site. The generators performed well through Sept. 11 and 12. But as the fuel supply dwindled, security-related restrictions on fuel trucks in the city left Telehouse and other carrier hotel owners scrambling to keep the generators running.
On Sept. 13, after talks with the Port Authority of NY & NJ, Telehouse secured a truckload of generator fuel with police escort through the Holland Tunnel. “We certainly got to the front of the line (for fuel shipments),” said Shields. “We have some very important customers, and they used their influence, especially EDS,” which uses the facility to clear transactions for the European financial system.
But soon after refueling, the generator overheated. On Sept. 14 a street generator from Con Edison was brought online, but soon ran out of fuel. That meant the building – whose tenants include EDS, Internap, Globix, Digital Island and Akamai – without power for much of Sept.14 and parts of Sept. 15.
The facility was able to use a Con Edison replacement generator while its in-house generator was repaired, and the building operated uneventfully on generator power from Sept. 15 until Con Ed restored power to the area on Sept. 26.
Telehouse said an immediate concern is assisting staff who continued working long hours while the rest of America was struggling to process the traumatic events of Sept. 11.
“We tried to be supportive (to our staff) any way we could,” said Anne Cecio, Telehouse Director of Human Resources. “During the worst of it, there was no way to give them the professional consolation they might have needed and we’re addressing that now. Witnessing the horror, and then continuing to work so closely to it, has impacted some of them.”
Telehouse said the extraordinary circumstances helped the company’s staff, customers and suppliers pull together during what normally might be a tense situation.
“Throughout the attack and the difficult days that followed, the outpouring of compassion and support from our clients held Telehouse together,” said Shields.
The economic fallout from 9-11 had a profound impact on the industry, accelerating existing problems associated with the “dot-bomb” bust. On Sept. 26, less than three weeks after the attack, industry leader Exodus Communications filed for bankruptcy.
A month after the attacks, I wrote a series of stories examining the impact of Sept. 11 changed the data center industry. They provide a look back at how leading industry observers assessed the future of the data center industry after a month of events that “left even industry veterans somewhat shell-shocked and wondering where the problems end and the recovery begins.”
[...] Sept. 11, 2001: One Data Center’s Story (datacenterknowledge.com) [...]