-
Codero Addresses Lengthy Power Outage
Dedicated hosting company Codero suffered a major power outage in its Phoenix data center early Monday that disrupted operations for several hours and caused lengthier downtime for about 10 percent of customers whose servers failed to restart properly. The incident began at about 8 a.m. Central time, when the facility lost utility power. The backup generators started properly, but an automatic transfer switch (ATS) failed to switch the power to generator power, leaving the data center operating on the battery banks of its uninterruptible power supply (UPS) units. “Unfortunately, time ran out and our facility went dark,” said Codero chief operating officer Ryan Elledge. The outage also damaged a power distribution unit (PDU) that supported the core network router, which delayed resumption of service after power was restored to the data center. A small number of servers remained offline late Monday evening due to hardware problems associated with the power issue. Codero staff provided updates and customer service throughout the day via the company’s Twitter channel, while Elledge provided a video update:
About Rich Miller
Rich Miller is the founder and editor-in-chief of Data Center Knowledge, and has been reporting on the data center sector since 2000. He has tracked the growing impact of high-density computing on the power and cooling of data centers, and the resulting push for improved energy efficiency in these facilities.
Tuesday data center tidbits. « The Server Room
Posted March 16th, 2010[...] up the story about the lengthy Cordero data center power outage yesterday. Sounds like great customer response with the use of Twitter, not so hot for not dealing with (or [...]
Inbred Texan
Posted May 1st, 2010Howdy I’m glad this was explained to us folks it effected most. There were many folks stranded in the real world and couldn’t log into their favorite places. Second Life for instance was down for many hours and its millions of users plumeted into the darkness. I’m glad that you seem to have discovered the root of the problem. I will assume this means we won’t have to relive such a traumatizing series of events again. 
Inbred Texan
Posted May 1st, 2010Wait a minute here!!! This says it happened on the 15th. Well I can tell you this, it happened again around the 28th for several hours. At least thats what residents and members of the virtual social/business network Second Life have been told. I am curious to see if these are lessons learned, or repeated mistakes that will continue to effect millions of people.
Circe Broom
Posted May 6th, 2010March 15th? Whoa! We were told that the huge blackout of Second Life was caused by the Phoenix data center’s power outage on April 28th-29th! Did it happen again? How is that possible? I am amazed.
unPC
Posted May 12th, 2010The COO, in explaining the March 15 failure, says that his people had to “replace some breakers” to fix the “automatic transfer switch.”
Most UL 1008 labeled transfer switches do not utilize breakers, although some do. It almost sounds like Codero just had a couple incoming breakers that were supposed to open and close to utility and generator power. Does anyone know this?
The transfer switch for a 24/7 facility should not only have a UL 1008 label, it should have a bypass-isolation feature as well. And it should be tested monthly: http://www.ecmweb.com/ar/electric_test_transfer_switch
Building A Cloud-Savvy Model for TCO and ROI
How Storage is Shaping The Cloud Data Center
Bringing Colo to the Customer: Modular Gets Local
Microsoft’s $1 Billion Data Center
March 16th, 2010