Comments on: Brief Power Outage for Amazon Data Center http://www.datacenterknowledge.com/archives/2009/12/10/power-outage-for-amazon-data-center/ News and analysis about data centers, cloud computing, managed hosting and disaster recovery Mon, 13 Feb 2012 17:24:17 +0000 hourly 1 http://wordpress.org/?v= By: vinod tomar http://www.datacenterknowledge.com/archives/2009/12/10/power-outage-for-amazon-data-center/comment-page-1/#comment-11444 vinod tomar Mon, 01 Mar 2010 09:42:24 +0000 http://www.datacenterknowledge.com/?p=19526#comment-11444 hi In the above case how should we operate the parallel redundant system so that cascade failures do not occur in the case of failure of one of the paths Thanks hi
In the above case how should we operate the parallel redundant system so that cascade failures do not occur in the case of failure of one of the paths
Thanks

]]> By: Reported cloud outages for Amazon, Google, Microsoft and Salesforce.com in 2008 and 2009 « Muon Cloud http://www.datacenterknowledge.com/archives/2009/12/10/power-outage-for-amazon-data-center/comment-page-1/#comment-10439 Reported cloud outages for Amazon, Google, Microsoft and Salesforce.com in 2008 and 2009 « Muon Cloud Sun, 31 Jan 2010 15:50:47 +0000 http://www.datacenterknowledge.com/?p=19526#comment-10439 [...] datacenterknowledge [...] [...] datacenterknowledge [...]

]]> By: LogicMonitor Blog » Active/Active or N+1? http://www.datacenterknowledge.com/archives/2009/12/10/power-outage-for-amazon-data-center/comment-page-1/#comment-9390 LogicMonitor Blog » Active/Active or N+1? Mon, 21 Dec 2009 22:52:40 +0000 http://www.datacenterknowledge.com/?p=19526#comment-9390 [...] The risk with Active/Active is that load does not scale linearly.  If you have two systems running at 40% load, that does not mean that one will be able the handle the load of both, and run at 80%.  More likely you will run into an inflection point, where you will run into an unanticipated bottleneck – be it CPU, memory bandwidth, disk IO, or some system that is providing external API resources. It can even be the power system. If servers have redundant power supplies, and each PSU is attached to separate Power Distribution Units (PDUs), the critical load for each PDU is now 40% of the rating.  If one circuit fails, all load switches to the other PDU – and if that PDU is now asked to carry more than 80% of its rating, overload circuits will trip, leading to a total outage.  There is some speculation that a cascading failure of this type was behind the recent Amazon EC2 outage. [...] [...] The risk with Active/Active is that load does not scale linearly.  If you have two systems running at 40% load, that does not mean that one will be able the handle the load of both, and run at 80%.  More likely you will run into an inflection point, where you will run into an unanticipated bottleneck – be it CPU, memory bandwidth, disk IO, or some system that is providing external API resources. It can even be the power system. If servers have redundant power supplies, and each PSU is attached to separate Power Distribution Units (PDUs), the critical load for each PDU is now 40% of the rating.  If one circuit fails, all load switches to the other PDU – and if that PDU is now asked to carry more than 80% of its rating, overload circuits will trip, leading to a total outage.  There is some speculation that a cascading failure of this type was behind the recent Amazon EC2 outage. [...]

]]> By: Julius Neudorfer http://www.datacenterknowledge.com/archives/2009/12/10/power-outage-for-amazon-data-center/comment-page-1/#comment-9062 Julius Neudorfer Fri, 11 Dec 2009 20:12:20 +0000 http://www.datacenterknowledge.com/?p=19526#comment-9062 The underlying question is did the "redundant PDU" actually fail because there was an unknown defect, or was it simply a case of a classic "Cascade Failure". This is when the total power, which is usually split 50%-50% across the 2 paths, exceeds the max rating of the power path i.e. 120 KVA split across 2 x 100 KVA paths. (each side sees only a 60% load [60 kva] so it "seems" ok) When a path fails (for any reason) the entire load shifts to the remaining path and it overloads and drops off line. The hence classic Cascade Failure. See http://www.eweek.com/c/a/IT-Infrastructure/How-to-Avoid-a-Redundant-Path-to-Power-Failure/ The underlying question is did the “redundant PDU” actually fail because there was an unknown defect, or was it simply a case of a classic “Cascade Failure”.

This is when the total power, which is usually split 50%-50% across the 2 paths, exceeds the max rating of the power path i.e. 120 KVA split across 2 x 100 KVA paths. (each side sees only a 60% load [60 kva] so it “seems” ok) When a path fails (for any reason) the entire load shifts to the remaining path and it overloads and drops off line.

The hence classic Cascade Failure.

See
http://www.eweek.com/c/a/IT-Infrastructure/How-to-Avoid-a-Redundant-Path-to-Power-Failure/

]]> By: Jeff http://www.datacenterknowledge.com/archives/2009/12/10/power-outage-for-amazon-data-center/comment-page-1/#comment-9050 Jeff Fri, 11 Dec 2009 14:56:25 +0000 http://www.datacenterknowledge.com/?p=19526#comment-9050 Apparently the future isn't here just yet. Isn't the point of "cloud computing" to allow redundancy at the server level so that any component failure can be corrected seamlessly (or at least in near-realtime) because some other part of the cloud is still operating? It seems to me that if EC2 isn't structured like this, then it is nothing more than managed hosting, the same old thing that's been popular for over a decade. Wake me up when "the cloud" is an ether of compute resources that can seamlessly tolerate a loss of any number of nodes up to the capacity threshold. Until then, "what's old is new again" is in full effect at EC2. Apparently the future isn’t here just yet.

Isn’t the point of “cloud computing” to allow redundancy at the server level so that any component failure can be corrected seamlessly (or at least in near-realtime) because some other part of the cloud is still operating? It seems to me that if EC2 isn’t structured like this, then it is nothing more than managed hosting, the same old thing that’s been popular for over a decade.

Wake me up when “the cloud” is an ether of compute resources that can seamlessly tolerate a loss of any number of nodes up to the capacity threshold. Until then, “what’s old is new again” is in full effect at EC2.

]]> By: Ernie http://www.datacenterknowledge.com/archives/2009/12/10/power-outage-for-amazon-data-center/comment-page-1/#comment-9043 Ernie Fri, 11 Dec 2009 13:08:30 +0000 http://www.datacenterknowledge.com/?p=19526#comment-9043 All too often a data center becomes inadequate immediately after it is build. Data Center engineers get complacent with their corner of the world as long as everything is operating correctly. I’m sure an autopsy will be performed in a conference room somewhere discussing what could have or should have been. Terms such as “We could not have known the second path would fail” or it wasn’t our fault that the second PDU went down. I have to point my finger at Amazon’s maintenance program. I will bet the preverbal dime to a donut that all their UPS, PDU, air conditioning and generator maintenance is up to date. I will then double down my bet by saying I bet they haven’t completed an electrical assessment of their facility since it was built. With the life cycle of a data center that begins to reach it's half-life six months after it was built, staying on top of the electrical infrastructure is paramount. To many times I have seen where “My UPS is running fine, I don’t need to worry about anything else” attitude. Once you begin to move around servers, you have changed the physical characteristics of the data center. NFPA 70E says you have to perform a circuit breaker coordination study every 5 years or after any changes. Things change daily in a data center and those changes need to be tracked. Data Center software such as aperture does this. If you are not willing to invest the money for a program such as that then you better invest money in a full time staff to do nothing but monitor changes to your electrical infrastructure. I have preached the gospel of “Just because it is new does not mean it works” my entire life. If there is a data center that has not completed an electrical assessment then that data center is a ticking time bomb. Time to lose the “UPS is running fine mentality” and pay attention to the electrical infrastructure of each and every data center. Cloud computing will not become a sky scraper as long as the foundation is built in sand. All too often a data center becomes inadequate immediately after it is build. Data Center engineers get complacent with their corner of the world as long as everything is operating correctly. I’m sure an autopsy will be performed in a conference room somewhere discussing what could have or should have been.

Terms such as “We could not have known the second path would fail” or it wasn’t our fault that the second PDU went down. I have to point my finger at Amazon’s maintenance program. I will bet the preverbal dime to a donut that all their UPS, PDU, air conditioning and generator maintenance is up to date. I will then double down my bet by saying I bet they haven’t completed an electrical assessment of their facility since it was built.

With the life cycle of a data center that begins to reach it’s half-life six months after it was built, staying on top of the electrical infrastructure is paramount. To many times I have seen where “My UPS is running fine, I don’t need to worry about anything else” attitude. Once you begin to move around servers, you have changed the physical characteristics of the data center. NFPA 70E says you have to perform a circuit breaker coordination study every 5 years or after any changes.

Things change daily in a data center and those changes need to be tracked. Data Center software such as aperture does this. If you are not willing to invest the money for a program such as that then you better invest money in a full time staff to do nothing but monitor changes to your electrical infrastructure. I have preached the gospel of “Just because it is new does not mean it works” my entire life.

If there is a data center that has not completed an electrical assessment then that data center is a ticking time bomb. Time to lose the “UPS is running fine mentality” and pay attention to the electrical infrastructure of each and every data center. Cloud computing will not become a sky scraper as long as the foundation is built in sand.

]]> By: The data center, the cloud, and power failure. « The Server Room http://www.datacenterknowledge.com/archives/2009/12/10/power-outage-for-amazon-data-center/comment-page-1/#comment-9024 The data center, the cloud, and power failure. « The Server Room Fri, 11 Dec 2009 04:18:47 +0000 http://www.datacenterknowledge.com/?p=19526#comment-9024 [...] } Today we got the story about Amazon’s outage of part of its EC2 cloud. In this post, I’ll examine what happened and what you can do to avoid the same [...] [...] } Today we got the story about Amazon’s outage of part of its EC2 cloud. In this post, I’ll examine what happened and what you can do to avoid the same [...]

]]>