NSA Data Center Plagued by Electrical Problems
October 8th, 2013 By: Rich Miller
The National Security Agency’s massive data center in Utah has been plagued by electrical problems that have delayed the opening of the 1 million square foot campus. The Wall Street Journal reports that there have been 10 incidents of arc flash “meltdowns” that have damaged equipment and delayed efforts to bring the facility’s power infrastructure online.
Citing project documents, the Journal reports that the failures have been the focus of more than 50,000 man hours of investigation and troubleshooting by contractors. The events have also caused significant equipment damage, incurring costs of up to $100,000 per incident.
An arc flash is an electrical explosion that generates intense heat that can reach 35,000 degrees Fahrenheit, which can damage and even melt electrical equipment. Arc flash incidents also represent a significant threat to worker safety.
Series of “Meltdowns”
There have been 10 arc fault “meltdowns” in the past 13 months at the NSA data center in Bluffdale, Utah, according to Journal. The electrical challenges at the facility were confirmed by the NSA and the construction team, a joint venture between Balfour Beatty Construction, DPR Construction and Big-D Construction Corp. The architectural firm KlingStubbins designed the electrical system.
“Problems were discovered with certain parts of the unique and highly complex electrical system,” the joint venture said in a statement. “The causes of those problems have been determined and a permanent fix is being implemented.”
NSA spokeswoman Vanee Vines told the paper that “the failures that occurred during testing have been mitigated. A project of this magnitude requires stringent management, oversight, and testing before the government accepts any building.”
Digital Juggernaut? Or Complex System?
As the NSA’s data collection efforts have made headlines in recent months, the Utah data center has become a symbol of the agency’s technology aspirations, taking on the aura of a digital powerhouse to house the nation’s intelligence secrets. The latest reports serve as a reminder that data centers are complex facilities that must be thoroughly tested to ensure safe operations.
The first arc fault failure at the Utah site occurred in August, 2012, according to project documents, with the most recent occurring on Sept. 25. More than 30 independent experts have conducted 160 tests over 50,000 man-hours, according to the Journal.
“Backup generators have failed numerous tests … and officials disagree about whether the cause is understood,” the paper wrote. “There are also disagreements among government officials and contractors over the adequacy of the electrical control systems, a project official said, and the cooling systems also remain untested.”
The NSA has said it will spend up to $1.5 billion on the Utah data center, which is approaching completion of its first phase after nearly four years of construction. The project will have a power capacity of 65 megawatts, making power a big component of its operations. The 1 million square-foot Camp Williams facility in Bluffdale, Utah will house 100,000 square feet of data center space, while the remaining 900,000 SF will be used for technical support and administrative space.
Hopefully no one was killed in the arc flashes and I hope they get the issues ironed out quickly.
It’s crazy how much power is going into a facility like that. One of a kind facility.
Another MikePosted October 13th, 2013
They call this a Massive Data Center, yet of the 1 million square foot campus they are only expecting to use 100,000 sqFT for actual data center, and the other 90% for “Tech Support and Admin”?? Doesn’t sound like a very massive data center, sounds like a huge spend of tax dollars for glorified offices. I can guarantee that Google or Apple are getting a ton more data center space for every $1.5 billion they spend. Isn’t this a concern to any tax payers?
joePosted October 14th, 2013
No one concerned this data center will be used to gather citizens digital records?
1) After 150 tests and 50000 man-hours, did any of these morons ever think of changing to a more reliable system?
2) Why do we know ANY OF this stuff? What moron gave them a picture of the place? The NSA has the highest security level, because it protects the lesser classifiied projects. The only thing these clowns didn’t do was hand out copies of their phone directory.
3) 9:1, serviced:servicing is actually a low ratio for a computer center. Which means this is also going to be an analysis shop.
4) WHO LET THEM KNOW ALL THIS?