Dublin Utility: Power Outage Not Caused by Lightning Strike

7 comments

The local power company in Dublin, Ireland now says a lighting strike was not the cause of the utility power outage that triggered downtime for major data centers operated by Amazon and Microsoft. The Electricity Supply Board (ESB) said the outage remained under investigation, but was caused by an equipment failure rather than lightning.

The power company issued its statement after readers in the comments of our post and in other media outlets raised questions  about the utility’s initial finding that lightning caused the outages -  an explanation which Amazon and Microsoft then shared in status updates for users and media.

ESB Networks confirms that it suffered a failure in a 110kV transformer in City West, Dublin at 6:16 p.m. local time on Sunday, August 7. “The cause of this failure is still being investigated at this time but our initial assessment of lightning as the cause has now been ruled out,” EBS Networks said in a statement. “This initial supply disruption lasted for approximately 1 hour as ESB Networks worked to restore supply. There was an ongoing partial outage in the area until 11pm.  The interruption affected about 100 customers in the Citywest area, including Amazon and a number of other data centres.  Another Amazon data centre served by ESB in South County Dublin was not directly affected by the outage, though it did experience a voltage dip which lasted for less than one second.”

The one-second voltage dip had been cited in a separate media report which added to the confusion about Sunday’s events.

ESB made it clear that in referencing a lightning strike, Amazon was sharing its best information at the time.

“Amazon accurately reported the information which had been passed to them from workers at the site,” said Marguerite Sayers, Head of Asset Management at ESB Networks. “Both the explosion and fire were localized to the bushings or insulators of the transformer and did not require the attendance of the emergency services. The extent of associated internal damage to the transformer was serious and resulted in supply interruption to a number of customers, and also impacted Amazon’s systems, as they have reported.”

Advanced data centers like the Amazon and Microsoft facilities usually continue operating through utility outages, relying on backup diesel generators to supply power. Both companies have said that fluctuations in the incoming utility power interfered with its automated systems to start backup generators once the power failed.

“When the transformer failed, there was a voltage dip experienced by customers in parts of Dublin for the duration of the fault,” Sayers said, who said she couldn’t comment on individual customers’ UPS or standby generator performance. “However, I can certainly confirm that this was an unexpected fault situation, with absolutely no advance warning, which did result in a voltage dip for many customers, in addition to supply loss for approximately 100 customers.”

About the Author

Rich Miller is the founder and editor at large of Data Center Knowledge, and has been reporting on the data center sector since 2000. He has tracked the growing impact of high-density computing on the power and cooling of data centers, and the resulting push for improved energy efficiency in these facilities.

Add Your Comments

  • (will not be published)

7 Comments

  1. Curt Gibson

    I can hardly wait to learn from this. The suspense is killing me.

  2. ‎"Both companies (Amazon & Microsoft) have said that fluctuations in the incoming utility power interfered with its automated systems to start backup generators once the power failed". Not much of a backup system if it needs the utility power to get it going. Methinks the warning system which triggers the backup generators needs a little tweaking!

  3. Mark is correct, Phasing issues or intermittent power from the untility should have nothing to do with the ability of the back-up systems to come on line, obviosuly the system needs a re-think of its design and modifications to make it more reliable...KT How Much does downtime cost? see note below from Data center providers The subsection entitled “Quantifying the Cost of Downtime” is particularly enlightening and examines nine categories of vulnerability. It calculates a loss of $5.600.00 per minute with extrapolated numbers that are frightening.

  4. Paul Nagy

    Best Practices need to be questioned here, with that massive storm, or isolated severe lightening in the area, a closed transition to generators until the threat is gone? Typically, most critical facilities have the detection systems in place, no mater what type of lightening protection system they have, all your generator feeds are typically underground and less likely to experience a fault, if anything, a strike that big, hitting the ground near the feeders, would be picked up by the UPS and gone to batteries or internal bypass. In which case, you would hope that the generator power system was not affected, in most cases, your worse case is being on static bypass on generators, had you performed a pre-transfer to gens. That's the best place to end up, not stuck on utility bypass during a severe thunder storm. What's the cost of diesel fuel vs downtime.