-
Dublin Utility: Power Outage Not Caused by Lightning Strike
The local power company in Dublin, Ireland now says a lighting strike was not the cause of the utility power outage that triggered downtime for major data centers operated by Amazon and Microsoft. The Electricity Supply Board (ESB) said the outage remained under investigation, but was caused by an equipment failure rather than lightning.
The power company issued its statement after readers in the comments of our post and in other media outlets raised questions about the utility’s initial finding that lightning caused the outages - an explanation which Amazon and Microsoft then shared in status updates for users and media.
ESB Networks confirms that it suffered a failure in a 110kV transformer in City West, Dublin at 6:16 p.m. local time on Sunday, August 7. “The cause of this failure is still being investigated at this time but our initial assessment of lightning as the cause has now been ruled out,” EBS Networks said in a statement. “This initial supply disruption lasted for approximately 1 hour as ESB Networks worked to restore supply. There was an ongoing partial outage in the area until 11pm. The interruption affected about 100 customers in the Citywest area, including Amazon and a number of other data centres. Another Amazon data centre served by ESB in South County Dublin was not directly affected by the outage, though it did experience a voltage dip which lasted for less than one second.”
The one-second voltage dip had been cited in a separate media report which added to the confusion about Sunday’s events.
ESB made it clear that in referencing a lightning strike, Amazon was sharing its best information at the time.
“Amazon accurately reported the information which had been passed to them from workers at the site,” said Marguerite Sayers, Head of Asset Management at ESB Networks. “Both the explosion and fire were localized to the bushings or insulators of the transformer and did not require the attendance of the emergency services. The extent of associated internal damage to the transformer was serious and resulted in supply interruption to a number of customers, and also impacted Amazon’s systems, as they have reported.”
Advanced data centers like the Amazon and Microsoft facilities usually continue operating through utility outages, relying on backup diesel generators to supply power. Both companies have said that fluctuations in the incoming utility power interfered with its automated systems to start backup generators once the power failed.
“When the transformer failed, there was a voltage dip experienced by customers in parts of Dublin for the duration of the fault,” Sayers said, who said she couldn’t comment on individual customers’ UPS or standby generator performance. “However, I can certainly confirm that this was an unexpected fault situation, with absolutely no advance warning, which did result in a voltage dip for many customers, in addition to supply loss for approximately 100 customers.”
About Rich Miller
Rich Miller is the founder and editor-in-chief of Data Center Knowledge, and has been reporting on the data center sector since 2000. He has tracked the growing impact of high-density computing on the power and cooling of data centers, and the resulting push for improved energy efficiency in these facilities.
Curt Gibson
Posted August 10th, 2011I can hardly wait to learn from this. The suspense is killing me.
”Both companies (Amazon & Microsoft) have said that fluctuations in the incoming utility power interfered with its automated systems to start backup generators once the power failed”. Not much of a backup system if it needs the utility power to get it going. Methinks the warning system which triggers the backup generators needs a little tweaking!
Les clouds d’Amazon et Microsoft à nouveau perturbés (màj) | Exanders.fr
Posted August 11th, 2011[...] un transformateur qui aurait subi une panne, entraînant la chute des deux datacenters. Selon Data Center Knowledge, l’enquête est toujours en cours pour déterminer les circonstances de la [...]
Mark is correct, Phasing issues or intermittent power from the untility should have nothing to do with the ability of the back-up systems to come on line, obviosuly the system needs a re-think of its design and modifications to make it more reliable…KT
How Much does downtime cost? see note below from Data center providers
The subsection entitled “Quantifying the Cost of Downtime” is particularly enlightening and examines nine categories of vulnerability. It calculates a loss of $5.600.00 per minute with extrapolated numbers that are frightening.
Paul Nagy
Posted August 13th, 2011Best Practices need to be questioned here, with that massive storm, or isolated severe lightening in the area, a closed transition to generators until the threat is gone? Typically, most critical facilities have the detection systems in place, no mater what type of lightening protection system they have, all your generator feeds are typically underground and less likely to experience a fault, if anything, a strike that big, hitting the ground near the feeders, would be picked up by the UPS and gone to batteries or internal bypass. In which case, you would hope that the generator power system was not affected, in most cases, your worse case is being on static bypass on generators, had you performed a pre-transfer to gens. That’s the best place to end up, not stuck on utility bypass during a severe thunder storm. What’s the cost of diesel fuel vs downtime.
To Cloud or Not to Cloud: What are Some of the Current, Most Pertinent Pros and Cons? « Ogalaws
Posted December 27th, 2011[...] Still on the subject of power supplies, a utility company outage in Dublin, Ireland, on August 7, 2011, first caused a service disruption in the cloud offerings of both Amazon and Microsoft, which have established significant data center facilities in that jurisdiction. Ordinarily, backup generators would have taken-over and immediately started to supply power. However, due to the strange nature of the outage – which a number of parties including both Microsoft and Amazon had originally and erroneously blamed on a lightning strike – their emergency backup system failed. See Rich Miller. Data Center Knowledge. Dublin Utility: Power Outage Not caused by Lightning Strike. Published on August 10, 2011. Available at: http://www.datacenterknowledge.com/archives/2011/08/10/dublin-utility-power-outage-not-caused-by-lig… [...]
Microsoft Azure Outage Explanation Doesn’t Soothe | courtneyallenbernard.com
Posted March 1st, 2012[...] only problem was the utility concluded three days later there had been no lightning strike. It said instead there had been an unexplained equipment [...]
Building A Cloud-Savvy Model for TCO and ROI
How Storage is Shaping The Cloud Data Center
Bringing Colo to the Customer: Modular Gets Local
Microsoft’s $1 Billion Data Center
August 10th, 2011