Outage in Dublin Knocks Amazon, Microsoft Data Centers Offline

[...] Ayer, un rayo que cayó en las afueras de Dublín provocó un incidente en los transformadores que abastec… y con los que dan servicio, en el caso de Amazon, a su plataforma de cloud computing EC2 y, en el [...]

paulz

Posted August 8th, 2011

We use an Amazon cloud based order management software (linnworks), it is down yesterday (7 Aug 2011) around 6PM, it is now 7AM 8 Aug 2011, it is still not working (the software can not find any available cloud server).

[...] un rayo que cayó en las afueras de Dublín provocó un incidente en los transformadores que abastec… y con los que dan servicio, en el caso de Amazon, a su plataforma de cloud computing EC2 y, en el [...]

[...] (ang. cloud) Amazon’u. Ucier­piał również sprzęt Microsoft’u. A to wszystko przez burzę w Dublinie i feralny piorun oraz wywołany nim pożar w okolicy, który położył na łopatki [...]

person287

Posted August 8th, 2011

That’s pretty bad, but I didn’t actually notice it on any of the sites or services I use.

[...] Lightning in Dublin Knocks Amazon, Microsoft Data Centers Offline (Data Center Knowledge) [...]

[...] Lightning in Dublin Knocks Amazon, Microsoft Data Centers Offline [...]

Soluweb

Posted August 8th, 2011

Those are some of the challenges Cloud computing is facing, many collateral damages.

Joerg Steegmueller

Posted August 8th, 2011

There are definitely odd discrepancies between the apparent facts and the stories Amazon and Microsoft have released:

There was no lightning strike yesterday in Dublin. The local power supply company had a short and very isolated outage in one small part of Dublin. There was NO explosion and no fire according to the local power supplier!

Not sure where Amazon got their information from and surprising that they can claim that their backup generators didn’t start because of an explosion if no explosion did take place.

Microsoft’s new and modern data centre is not in the area where the power outage happened, and due to the limited information available from Microsoft it can not be concluded that it happened at the same time!

Yes, there was a very short (less than a second) power outage and the local electricity supplier says that the power source was immediately switched over to an alternate supply. But there seems to be some issues in Amazon! Otherwise their backup systems would have kicked in properly!

Skaperen

Posted August 8th, 2011

“Power sources must be phase-synchronized before they can be brought online to load.”

That’s only true if you are trying to make a closed-transition switchover (whether mains is live or dead). A closed-transition system is useless for maintaining power from generators, since the mains outage can come as a surprise faster than even the fastest startup generators can kick in. The solution is to use open-transition switching to generators supplying a datacenter that is fully protected by a UPS/battery infrastructure (whether one per datacenter, per row, per rack, or per machine). Stagger the switchover in groups or section the generators for a smoother transition. Don’t even try to parallel generators in the design, just section everything apart by generator/switch.

DG

Posted August 8th, 2011

Skaperen,

I think they mean that they had to parallel the generators together onto one bus, not paralleling the utility power and the generators. i.e. bring up one generator, bring up the second, watch synchroscope go around, and hit the breaker when it gets to 12 oclock.

They had UPS. They just didnt have a good staff to manually bring up the gensets before the UPS power died. Still, a better design would have prevented the downtime.

[...] service and left it unable to start its generators, resulting in a total power outage, according to Data Center Knowledge, which originally reported the [...]

[...] by the same lightning strikes in Dublin that affected Amazon’s datacenter was explained in a Data Center Knowledge story. Microsoft referred customers to private dashboards used by Business Productivity Online Service [...]

[...] Center Knowledge reported this week that lightning knocked out major cloud computing data centers in the Dublin [...]

[...] It affected one Availability Zone (AZ) in AWS Europe. Rich Miller of Data Center Knowledge has detailed information on the incident.  Amazon said that lightning struck a transformer near its data center, [...]

[...] this article: Lightning in Dublin Knocks Amazon, Microsoft Data Centers Offline – Data Center Knowledge Related [...]

Henrik

Posted August 10th, 2011

it is just to much they dont have enough control of back up procedures and so on

Rich Miller

Posted August 10th, 2011

The local utility has now confirmed that lightning was not the cause of the transformer failure that led to the outage. We’ve updated our story to reflect this. See this update for details:

http://www.datacenterknowledge.com/archives/2011/08/10/dublin-utility-power-outage-not-caused-by-lightning-strike/

Doug

Posted August 12th, 2011

It’s instances like these that have to remind all of us how vulnerable our industry can be. Something so small can have such a massive international effect.

[...] week’s possibly lightning-caused outages at Microsoft and Amazon Web Services reiterated a very important lesson in cloud computing: Stuff happens, and even the best-laid plans [...]

[...] Tech news. News gadgets reviews and secrets. Last week’s possibly lightning-caused outages at Microsoft and Amazon Web Services reiterated a crucial lesson in cloud computing: Stuff happens, or even [...]

[...] August 7, power equipment failures took a portion of Amazon’s EC2 cloud computing platform in Europe offline for 24-48 [...]

[...] snapshots, and back ups for when EBS fails. And cloud is made by humans. Trucks hit generators. Lighting strikes. And that means the cloud will fail and the customer may not have any recourse to have prevented [...]

[...] y centralizados que no sólo impulsan el control sino que por su lógica centralizada son, además poco robustos. Y por si quedaran dudas, los que realmente saben de esto, las dictaduras, no tienen dudas: quieren [...]

[...] electric deviation” and disabling some of a energy synchronization systems, according to a report by record site, Data Center Knowledge. Tags: malpractice, malpractice insurance, medical malpractice, professional liability [...]

[...] 212011   Selling Down The River is back online.  I lost everything due to a mixture of an issue at the Amazon Ireland Datacentre and my own incompetence about how EC2 works with snapshots and AMI’s.  It’s taken a [...]

[...] those with their own agenda tend to pile on — even when it’s a complete freak of nature like the lightning strike in Ireland last [...]

[...] DataCenterKnowledge report that this lightning strike has caused downtime for many sites using Amazon’s EC2 cloud computing platform, as well as users of Microsoft’s BPOS (Business Productivity Online Suite). In fact,  some sites that rely on one of its storage services took between 24 and 48 hours to be fully recover. [...]

[...] Some can be puzzling, like one in Dublin last year affecting both Amazon and Microsoft that Amazon initially said was caused by a lightning strike hitting a generator, leading to an explosion and fire. It turned [...]