Power Failure KOs Intuit Sites for 24 Hours

11 comments

A major site outage that knocked Intuit web sites offline for more than 24 hours was caused by a data center power failure during routine maintenance, the company said tonight. The downtime affected the Intuit.com web site and thousands of small business customers who host their websites on Quickbooks.com.

“Our preliminary investigation indicates the outage occurred during a routine maintenance procedure Tuesday night,” Intuit CIO Ginny Lee said on the company’s customer forum. ” An accidental power failure during that procedure affected both our primary and backup systems, taking a number of Intuit websites and services offline. While power was quickly restored, we’re working diligently to validate our systems and bring them back into full operation.”

As of 7:30 p.m. Pacific time Wednesday, Intuit said it was beginning to restore the web sites and services that had been offline since 7 p.m. Tuesday. The effort followed a long day of frustration for Intuit’s small business hosting customers, most of whom were unable to access their web sites or process transactions.

About the Author

Rich Miller is the founder and editor-in-chief of Data Center Knowledge, and has been reporting on the data center sector since 2000. He has tracked the growing impact of high-density computing on the power and cooling of data centers, and the resulting push for improved energy efficiency in these facilities.

Add Your Comments

  • (will not be published)

11 Comments

  1. Anon

    Is it really a great idea to publish the locations of any company's data centers? I was under the impression that most businesses prefer that the location of their data centers remain unadvertised and secret as to not bring any unwanted attention. Irresponsible reporting here in my opinion.

  2. Virus

    A power failure does not knock out primary and backup systems for this long!!! The site was attacked! Intuit does not want to admit it. Such a large scale system that holds personal info has complete redundancy built into it, unless a major security problem has taken place that enables someone to shut the system down. IF it is actually true that the system went down from power failure, Intuit needs to find a new CIO! Someone who knows how to build redundant systems and have contingency rules in place. Ginny Lee.....you screwed up!

  3. bks

    Ginny Lee knows nothing about computers and everything about speaking management gobbledygook. Intuit *is* an IT company. There is no excuse for this and heads should roll. I guess we now know what Ginny meant by "Lean IT"! --bks

  4. Aaron

    There are a lot of possibilities and to speculate Intuit was attacked or that big name companies always build redundant systems or manage their systems properly is naive. There are a lot of reasons why an outage can last longer with the primary reason being safety to personnel. People think you should just flip a switch and magically "restart" 480V gear. If something in the electrical system fails, if you value life and property you will be certain to identify what it is prior to just pushing that reset button. In the data center world, if something fails it usually did so for human or mechanical reasons. Secondly, maybe the power problem was minor but the data center personnel had to wait on vendor resources to check the equipment and resume normal power operations. It's also very possible that the facility folks did their job and then it was up to the IT resources to start equipment and databases backup and/or restore after a loss of power But I'm just speculating - the point is you don't need a new CIO because a data center has a 24 hour outage....You probably need a new maintenance plan, procedures, or equipment..

  5. Jeff

    They probably just need bigger "TURN LEFT FOR SOURCE A, RIGHT FOR SOURCE B" signs on their transfer switches... And maybe a remedial "Left vs Right" class for any operators of said switch. In all likelihood they were doing single side maintenance, had it de-energized and accidentally switched their load to it thanks to a careless operator. After a few operators soiled themselves, they returned the switches to "auto", opened the downstream breakers, then slowly re-energized the whole system. Even after closing all the breakers, there is still hours if not days worth of work to return every single server to operation. Most companies don't plan for graceful shutdown/restart since they have many layers of redundancy (and 5 days of fuel for their generator) and never expect to have to go without power. Underestimating the human ability to screw something up is a very dangerous mistake!

  6. I would tend to agree, a power outtage does not know out primary and back up systems. All professional web hosting systems have power backups so there is no interruption of service. I do not host my site with QuickBooks, but I do use them for credit card processing. Of which I could not do at all on Wednesday. When their system has had maintenance in the past, it is usually 20 to 30 minutes that it is down, not 36 hours. And truthfully, Intuit does not have to tell us what caused the problem. They have the right to say "We are experiencing technical difficulties." So we can conjecture all we want, and we know something is wrong, and we will go on living just fine not even knowing what happened. And a month from now, just about nobody will remember....

  7. bks

    There is no excuse for the lack of transparency. Even if it's nothing more than saying "there there, everything will be okay" they should communicate every few hours. Ginny Lee must go. --bks

  8. Cynthia

    I was patient through all of this. I have a store plus a website and couldn't process any transactions. For the store customers wanting to use a credit card, I simply took their information and processed it when Intuit was up and running. My big issue is that I called Quickbooks at 2:15 Wednesday afternoon and asked when they would be back up. The man on the phone told me it would be up in "exactly 43 minutes". I didn't get his name but he should be let go for lying.

  9. this was really unfortunate and a friend's site got affected. whatever the reasons were, I hope they make sure that this never happens again as it can be a huge blow to those who depend on their sites for their livelihood.