Comments on: When The Power Goes Out at Google http://www.datacenterknowledge.com/archives/2010/03/08/when-the-power-goes-out-at-google/ News and analysis about data centers, cloud computing, managed hosting and disaster recovery Mon, 13 Feb 2012 17:24:17 +0000 hourly 1 http://wordpress.org/?v= By: How not to handle a data center power outage, starring Google. « The Server Room http://www.datacenterknowledge.com/archives/2010/03/08/when-the-power-goes-out-at-google/comment-page-1/#comment-11990 How not to handle a data center power outage, starring Google. « The Server Room Fri, 12 Mar 2010 16:58:44 +0000 http://www.datacenterknowledge.com/?p=23290#comment-11990 [...] } Today’s post comes from the post mortem of the Feb 24 data center power outage that took down Google’s AppEngine cloud com.... The issue I see here is not the power outage itself or the fact that it wiped out such a large [...] [...] } Today’s post comes from the post mortem of the Feb 24 data center power outage that took down Google’s AppEngine cloud com…. The issue I see here is not the power outage itself or the fact that it wiped out such a large [...]

]]> By: Google reveals reasons for data centre problems | Tech Ticker http://www.datacenterknowledge.com/archives/2010/03/08/when-the-power-goes-out-at-google/comment-page-1/#comment-11840 Google reveals reasons for data centre problems | Tech Ticker Tue, 09 Mar 2010 22:31:25 +0000 http://www.datacenterknowledge.com/?p=23290#comment-11840 [...] of hours, it did paint a picture of what it would be like if its centres failed more often. The post-mortem of the power failure, which shut down the company’s mobile apps service, showed that lack of human intervention [...] [...] of hours, it did paint a picture of what it would be like if its centres failed more often. The post-mortem of the power failure, which shut down the company’s mobile apps service, showed that lack of human intervention [...]

]]> By: Google Apps Downtime Report: Perfect Example? « UNIX Administratosphere http://www.datacenterknowledge.com/archives/2010/03/08/when-the-power-goes-out-at-google/comment-page-1/#comment-11816 Google Apps Downtime Report: Perfect Example? « UNIX Administratosphere Mon, 08 Mar 2010 21:40:53 +0000 http://www.datacenterknowledge.com/?p=23290#comment-11816 [...] a near-perfect example of a good report. Data Center Knowledge summarized the event well in an article; they have also spoken with Google previously about how they handle [...] [...] a near-perfect example of a good report. Data Center Knowledge summarized the event well in an article; they have also spoken with Google previously about how they handle [...]

]]> By: Monday data center tidbits. « The Server Room http://www.datacenterknowledge.com/archives/2010/03/08/when-the-power-goes-out-at-google/comment-page-1/#comment-11813 Monday data center tidbits. « The Server Room Mon, 08 Mar 2010 20:13:04 +0000 http://www.datacenterknowledge.com/?p=23290#comment-11813 [...] up is the post mortem on the Google AppEngine power failure. It’s bad enough to have the power failure in the first place, but not to have anyone who [...] [...] up is the post mortem on the Google AppEngine power failure. It’s bad enough to have the power failure in the first place, but not to have anyone who [...]

]]> By: When the Power Goes Out At Google | JetLib News http://www.datacenterknowledge.com/archives/2010/03/08/when-the-power-goes-out-at-google/comment-page-1/#comment-11802 When the Power Goes Out At Google | JetLib News Mon, 08 Mar 2010 16:18:21 +0000 http://www.datacenterknowledge.com/?p=23290#comment-11802 [...] has issued an incident report on a Feb. 24 outage for Google App Engine, which went offline when an entire data center lost power. The post-mortem outlines what went wrong and why, lessons learned and steps taken, which include [...] [...] has issued an incident report on a Feb. 24 outage for Google App Engine, which went offline when an entire data center lost power. The post-mortem outlines what went wrong and why, lessons learned and steps taken, which include [...]

]]> By: Ernie http://www.datacenterknowledge.com/archives/2010/03/08/when-the-power-goes-out-at-google/comment-page-1/#comment-11796 Ernie Mon, 08 Mar 2010 14:10:34 +0000 http://www.datacenterknowledge.com/?p=23290#comment-11796 I'm glad to see Google step to the plate and institute reviews and run additional drills. Within the last few months I have witnessed a datacenter go dark because of a comedy of errors. 1. Utility outage 2. Data Center on UPS Battery 3. Generator Starts 4. Automatic Bypass Switch Fails 5. None of the Data Center personnel knew how to operate the Automatic Bypass Switch bypass breaker or Manual Override 6. UPS Battery reached End of Discharge (EOD) 7. Data Center Goes Dark 8. Event investigators found large sign on ATS bypass switch ( Do Not Operate) You must schedule outages to complete maintenance on all your circuit breaker, automatic transfer switches, UPS and generators. Train your staff to operate those bypass breakers, manual override on Automatic Transfer Switches and UPS static bypass. I’m glad to see Google step to the plate and institute reviews and run additional drills. Within the last few months I have witnessed a datacenter go dark because of a comedy of errors.

1. Utility outage
2. Data Center on UPS Battery
3. Generator Starts
4. Automatic Bypass Switch Fails
5. None of the Data Center personnel knew how to operate the Automatic
Bypass Switch bypass breaker or Manual Override
6. UPS Battery reached End of Discharge (EOD)
7. Data Center Goes Dark
8. Event investigators found large sign on ATS bypass switch
( Do Not Operate)

You must schedule outages to complete maintenance on all your circuit breaker, automatic transfer switches, UPS and generators. Train your staff to operate those bypass breakers, manual override on Automatic Transfer Switches and UPS static bypass.

]]>