Insulation Failure Cited in Fisher Plaza Outage
A major power outage that disrupted data centers at Seattle’s Fisher Plaza in July was probably caused by an insulation failure in an electrical bus duct that connected the building to the power grid, according to a report to tenants by the building’s owner.
The July 2 incident knocked payment processor Authorize.net offline, disrupting e-commerce for thousands of web sites, while also causing lengthy downtime for Microosft’s Bing Travel service, domain registrar Dotster, colocation company Internap and web hosting provider AdHost,
An investigation by Power Science Engineering Inc. of Shoreline, Wash. concluded that the failure of two bus ducts – enclosures housing copper bars to conduct electricity – in a basement power room of Fisher Plaza was likely caused by “progressive degradation” of insulation, which ultimately led the bus duct to fail. The incident report is available at the TechFlash web site.
No Single ‘Stimulus’
Investigators said such failures are usually prompted by a “stimulus” such as movement of the duct or sudden changes in large loads, but that none of those conditions existed at the time of the failure at 11:11 p.m. on July 2. Power Science concluded that the insulation wore down over time due to thermal and mechanical stresses at a bend in the duct.
The report rejected two other theories – that the bus duct was damaged by a falling light fixture or an accidental triggering of a sprinkler system. A flourescent light fixture was found laying on top of the bus duct, but investigators said the fixture collapse was caused by the “significant heat” from the electrical failure.
Water Damage Discounted
The report also found no evidence that the sprinkler system turned on prior to the electrical event, and noted that the bus duct was enclosed in a water-tight metal container. The initial report from the Seattle Fire Department said that “engineers suspect that water on the copper plates (of the bus) resulted in an arc and the subsequent fire damage.”
The Power Science report noted that “routine maintenance and testing” such as load data and infrared scanning can help detect problems with aging power infrastructure, although this is more difficult for enclosed components like bus bars inside ducts. Fisher Plaza officials told TechFlash they conducted regular infrared and thermal scans and physical inspection of the equipment.
Power Science Engineering also said bus ducts can be separated to limit damage from a failure, or replaced by cables. “In our opinion, use of cable systems at critical locations can be a better option to bus ways because the cable systems use insulated cables instead of bus bars,” the report said.
Building owner Fisher Communications has been powering Fisher Plaza using temporary generators while it rebuilds the core power infrastructure. The new design will use bus ducts, but will separate them to avoid simultaneous failures.
[...] } Just reading an article about the analysis of the Fisher Plaza data center fire. Here are some of my thoughts on the issue and prevention [...]
Michael D. FontainePosted September 19th, 2009
I have made some calculations based on simplifying assumptions and the Engineer’s report. It would appear that there is a transformer vault with four 2,000 kVA transformers connected together in a bus on the secondary side (480Y/277 V). It would appear that there is no secondary protection from the transformers to the collector bus or no secondary protection from the collector bus to the busduct, which is 5,000A. There are two 5,000A busducts connected to the collector bus in the transformer vault. Furthermore, there is no mention of expulsion or partial current limiting fuses being used in the transformers themselves on the primary side. The only protection mentioned on the primary side of the transformers is the power circuit breakers in the utility substation upstream. If the utility system is considered as an infinite bus, then the available fault current at the collector bus is as follows:
Rough Three-Phase Short Circuit Calculation (based on assumption of infinite utility bus and ANSI standard impedance (5.75% + 7.5% = 5.3% to 6.2%)
Fault current from one transformer IF = 2000/0.53 = 37, 736 A
The available fault current from four transformers IFTOTAL = 37,736A x 4 = 150,944A
The arcfash incident energy potential here has to be through the roof – not only is this a stiff system with a lot of capacity, it is a system that is relying on the upstream utility substation circuit breakers to trip, which extends the tripping time. As should be known, Arcflash incident energy depends on available current (I) and time to trip (t). Without performing any calculations, I believe that would be safe to say that there is no safe level of PPE for working on the main switchboard. This means that work should not be performed on the switchboard while it is live. This is the way electrical systems for large buildings were done in the late seventies, when arcflash was not part of the design considerations.
Since then, starting in and around 1995 Arcflash considerations started being considered in electrical designs. It is now the case (required by OSHA and NFPA 70E) that an electrical safety hazard analysis be performed. Arcflash is part of that analysis requirement.
Does the facility have an Electrical Safety Program as required by OSHA and have they performed an electrical safety analysis? If they have or if they do in the future, they may need to consider making significant adjustments to the electrical distribution system based on electrical safety considerations dictated by OSHA.