Car Crash Triggers Amazon Power Outage
May 13th, 2010 By: Rich Miller
Amazon’s EC2 cloud computing service suffered its fourth power outage in a week on Tuesday, with some customers in its US East Region losing service for about an hour. The incident was triggered when a vehicle crashed into a utility pole near one of the company’s data centers, and a transfer switch failed to properly manage the shift from utility power to the facility’s generators.
Amazon Web Services said a “small number of instances” on EC2 lost service at 12:05 p.m. Pacific time Tuesday, with most of the interrupted apps recovering by 1:08 p.m. The incident affected a different Availability Zone than the ones that experienced three power outages last week.
The sequence of events was reminiscent of a 2007 incident in which a truck struck a utility pole near a Rackspace data center in Dallas, taking out a transformer. The outage triggered a thermal event when chillers struggled to restart during multiple utility power interruptions.
Crash Triggers Utility Outage
“Tuesday’s event was triggered when a vehicle crashed into a high voltage utility pole on a road near one of our datacenters, creating a large external electrical ground fault and cutting utility power to this datacenter,” Amazon said in an update on its Service Health Dashboard. “When the utility power failed, most of the facility seamlessly switched to redundant generator power.”
A ground fault occurs when electrical current flows into the earth, creating a potential hazard to people and equipment as it seeks a path to the ground.
“One of the switches used to initiate the cutover from utility to generator power misinterpreted the power signature to be from a ground fault that happened inside the building rather than outside, and immediately halted the cutover to protect both internal equipment and personnel,” the report continued. “This meant that the small set of instances associated with this switch didn’t immediately get back-up power. After validating there was no power failure inside our facility, we were able to manually engage the secondary power source for those instances and get them up and running quickly.”
Switch Default Setting Faulted
Amazon said the switch that failed arrived from the manufacturer with a different default configuration than the rest of the data centers’ switches, causing it to misinterpret this power event. “We have already made configuration changes to the switch which will prevent it from misinterpreting any similar event in the future and have done a full audit to ensure no other switches in any of our data centers have this incorrect setting,” Amazon reported.
Amazon Web Services said Sunday that it is making changes in its data centers to address a series of power outages last week. Amazon EC2 experienced two power outages on May 4 and an extended power loss early on Saturday, May 8. In each case, a group of users in a single availability zone lost service, while the majority of EC2 users remained unaffected.
EdPosted May 13th, 2010
I’d like to know who commissioned this facility…
Was the transfer switch setting outside a desirable limit? Did Amazon startup the transfer switch equipment before cutting over to building load? Typically the install is commissioned before assuming building load to discover setting issues. Surprised Amazon would not have performed that check.
BobPosted May 13th, 2010
If there was a real UPS between the transfer switch and the load, then a 10 or 15 minute outage wouldn’t cause too much trouble. (Ditto if the load devices were wired to independent A/B power sources with separate upstream transfer switches and independent utility feeders.)
So what WAS between the transfer switch and the load? Nothing? One of those Hitec rotational-energy-storage units (with 15 seconds of run time)?
This is just about money. The guys who build the data centers never want to spend any more than absolutely necessary. They sleaze by with the minimum possible configurations, just like the BP oil platform in the gulf, and the single-hulled tanker ships and collapsed-ring SONET systems. Its always about money. Bigwigs pocket the fat bonus checks while the country takes the express elevator down to the fiery hot place below.
[...] writes “An Amazon cloud computing data center lost power Tuesday when a vehicle struck a nearby utility pole. When utility power was lost, a transfer switch in the data center failed to properly manage the [...]
Walter WhitePosted May 14th, 2010
I wonder who manufactured the switch.
ATS guyPosted May 14th, 2010
Do they use a UL 1008 listed contactor style transfer switch or a cheaper circuit breaker style?
Data Center guyPosted May 14th, 2010
In response to Bob, while I am not sure what type of UPS Amazon uses, they often are not the problem. When utility power is lost, so is cooling because most sites do not put large three phase motors on UPS power and that is what takes down many sites. Also, most UPS’s don’t have more than a 7 minute run time.
This is a technical forum, if you want to post about politics, i am sure you can find a bunch of like minded folks at the Washington Post.
NDLPosted May 14th, 2010
Wow. I thought redundancy took things like that out of the equation. So much for redundant this and redundant that.
TusharPosted May 14th, 2010
Can one of you data center guys provide some guidance?
How long does it usually take the standby generators to kick in fully (i.e. to power the CRAC) once a powe outtage occurs?
In that time frame, how much heat gain/load is to be expected, especially in these centers with high density blade server racks?
Would the increased heat load during this time frame result in damage to equipment? Have you seen this or more of a theoretical possibility?
Knowing the critical nature of the switch, its settings should have been compared to the existing switches to mitigate this issue ever occurring.
SuperDCPosted May 14th, 2010
Typically our generators kick in within 10 seconds of a power outage.
Friday data center tidbits: Amazon bozos #cloudcomputing (again)(and again and again) « The Server RoomPosted May 14th, 2010
[...] } Today I’ve been reading more detail into yet another recent data center power failure of the Amazon EC2 cloud computing service. There isn’t much they could do about having a car crash into a utility pole, but the rest of [...]
BobPosted May 14th, 2010
It just shows you that you can’t throw stuff in the cloud and think you have given up responsibility. You still have to think about disasters, prepare for them and test for them. Do you know how robust your cloud provider is?
[...] most recent affected Amazon EC2 users on 13 May. This outage was caused when a cutover from the loss of utility [...]
[...] (CSPs) consider their cloud infrastructure as bulletproof. Apparently this is not the case as this story shows. Amazon’s EC2 cloud computing service suffered its fourth power outage in a week on Tuesday, with [...]
HardwareFreakPosted May 16th, 2010
The problem here isn’t the switch supposedly not working properly. That’s a BS PR excuse to blame the vendor. The facility management team would have made sure all switches were properly set, unless they’re incompetent.
The real problem is above ground power which allows vehicles, storm systems, etc to damage AC infrastructure killing the feeds. Any modern data centers should be receiving all grid power from cables buried all the way from the substation to the facility. There is zero excuse for any data center in 2010 to be receiving power via overhead lines.
Knowing Amazon, I’d bet this facility does have buried cable service. It’s just too easy to blame the incident on a truck taking down a pole, so they did, knowing full well that the story is too small for any entity to do some investigative journalism. Their story (likely a falsehood) will stand until someone digs up traffic accident reports for that city on that day, and confirms whether a power pole was stricken.
[...] the article, “Amazon’s EC2 cloud computing service suffered its fourth power outage in a [...]
In my many years of building, commissionng and maintaining data centres, I must say, this is the sorriest excuse I have seen a data centre come up with when an outage has occurred.
Even if the ATS has ground fault sensing………and in the event this sensing “failed”, the other sensing features of the ATS would have started and transfered the generator. Every modern ATS monitors both voltage and frequency of utility power and in the event they fall outside the set parameters, will start the generator and transfer the site.
Will all that said, if Amazon sticks by this story, then they must admit that they didn’t commission the ATS’s and generator correctly as well as admit that they are not performing proper monthly maintainence of the back up system………..
Personnally, I think they should come clean with the real fault!!!
Roberto CazadorPosted May 19th, 2010
Can you please comment on why Amazon doesn’t have the same power outage protection as Terremark? See recent article:
Thanks in advance.
[...] customers in its US East Region losing service for about an hour. The incident was triggered when a vehicle crashed into a utility pole near one of the company’s data centres, and a transfer switch failed to properly manage the [...]
IanPosted September 15th, 2010
Unless they were supporting a hospital surgical unit, or military action, there isn’t much else that can’t survive a 1 hour outage. The ‘never down, ever” attitude only applies in life or death situations, which justifies the big bills. If someone’s phone app wasn’t working, or their social network was delayed, so what?
Amazon has to credit the cost back to the customer, it’s not a free ride for them when there’s an interruption. So lesson learned, correct for it and move on.
GazPosted October 15th, 2010
A lot of the comments here make me laugh.
Even with levels of refundancy, there are always occasions where downtime may happen over the course of several years.
Just goes to show, a lot of people like to think they know it all. If they worked in IT, they might understand the pressures and compexities of these systems.