“This morning, an Equinix data center experienced a power failure that affected a number of customers, including Zoho,” Equinix said in a statement. “We immediately notified the affected customers and full service was restored within a matter of seconds. We are investigating the root cause of the issue. We deeply regret the impact and inconvenience this incident caused our customers.”

Although power to the data center was restored within seconds, the sudden loss of power is problematic for database-driven applications, which means that short power outages can translate into hours of recovery time for services. That was the case for Zoho.

“Services have multiple database clusters – like 8-12 database clusters per each service,” Zoho explained in a blog post. “(The) power failure caused inconsistencies in some of these clusters. To restore these services fully, we need to make sure all these database clusters are consistent. This database consistency check and sync is what is taking time.”

Zoho reported the outage at 8:46 am Eastern, and began restoring services by 1:30 pm. As of 4:30 pm, the majority of services appeared to be back online. The company’s applications have more than 5 million users.

A graph of Black Friday activity at retail sites shows traffic peaking at midnight, indicating strong activity by mobile deal-seekers as well as "couch commerce" among shoppers seeking virtual doorbusters. (Source: Akamai)

Today is Cyber Monday, and all the trends suggest solid growth in online retail activity. Early data from the holiday weekend highlight three trends: strong overall year-to-year growth in online sales, the emergence of Thanksgiving evening as a major event for retail (both online and off) and the increasing use of mobile phones and tablets on retail sites. The growth in mobile traffic appears to include comparison shopping by deal-seekers in stores, as well as an increase in “couch commerce” by surfers seeking virtual doorbuster sales.

Some of the best data and insights came from the Akamai blog, which featured regular updates from Akamai e-commerce strategist Lelah Manz. Here’s our roundup of holiday retail data, featuring several informative posts from Akamai, as well e-commerce activity seen by IBM and Paypal:

• Akamai (Thanksgiving Day) – “With Akamai retail traffic up 76% on average, and 74% at peak at 9:00 PM EST, there were strong indications that Thanksgiving will emerge once again as the star of the holiday shopping weekend.  If we assume the same conversion index (see Part 2 of this holiday blog series) then Thanksgiving Day could exceed $700 million, surpassing Black Friday online totals last year.”
• Akamai (Black Friday) – “While nowhere near as explosive as Thanksgiving Day, the traffic growth continues to trend at much higher levels than 2010, on average 30% higher.  The open question from Thanksgiving was whether Thanksgiving Day was just a shift of shopping activity from Black Friday, or a new shopping day in its own right.  With Black Friday growth data so strong, it appears to be the latter.”
• IBM – “Black Friday witnessed the arrival of the mobile deal seeker who embraced their devices as a research tool for in-store and online bargains, according to data from IBM’s Smarter Commerce business. Mobile traffic increased to 14.3 percent on Black Friday 2011 compared to 5.6 percent in 2010, while sales on mobile devices surged to 9.8 percent from 3.2 percent year over year. Mobile shopping was led by Apple, with the iPhone and iPad ranking one and two for consumers shopping on mobile devices (5.4 percent and 4.8 percent respectively).”
• PayPal – PayPal documented the traffic spikes growth trend for Thanksgving day buying, as well as the mobile surfing. “There was a more than three-fold (350%) increase in the number of global customers shopping through PayPal mobile on Thanksgiving 2011 compared to last year, the company said. Thanksgiving 2011 resulted in a five-fold (511%) increase in global mobile payment volume compared to Thanksgiving 2010.”

Simply put the issue is indifference. Disaster recovery is about what might happen and when operational matters in the here and now need to be addressed then the dollars and manpower will flow to the need.  However that could be a costly mistake if an organization suffers an outage and doesn’t have a disaster recovery plan that has kept pace with the changes in the data center. For optimizing continuity, time and money need to be allocated for the management of three distinct disciplines:

• A continual update of the recovery process
• Maintaining the staff for test and recovery
• Building in the expertise to handle the recovery

As noted most organizations do not have the means to handle this additional process and expense so the answer is in outsourcing.  Besides minimizing the potential losses from a disaster an outsourced solution can also provide other value added benefits to the day to day operations.  In reviewing this whitepaper you will get a sense of how to minimize downtime and manage the disaster recovery lifecycle.

• BlackBerry-maker to look at setting up regional data centres – Speaking to Indian media via tele-conference, Mr Patrick Spence, Senior Vice-President and Managing Director, Global Sales & Regional Marketing, RIM, said, “Our immediate focus is on getting back the services to where we would like it to be. We will leave no stone unturned to make sure that this does not repeat. All options, including setting up network architecture with regional hubs, are on the table. If that makes sense, we will do it.” From the Hindu Business Line.
• BlackBerry blackout: how it happened – From the Telegraph: “RIM’s investigation revealed the apparent cause of the outage to be a failed Cisco switch in its core network. Switches are basic components of Internet Protocol networks. They are specialised computers that direct communications within networks; in this case the emails, web browsing and instant messages of millions BlackBerry Internet Service users. On day three of the crisis, RIM publicly admitted it had suffered a “core switch failure”. If everything had worked to plan, the failure would not have mattered. A backup system also failed, however, for reasons that remain obscure and will surely be among the top priorities of RIM’s own post-mortem investigation.
• RIM: No Network Changes Were Made – All ThingsD provided a liveblog of a call with RIM executives, who addressed a key question: “No recent changes were made. The system is provided by multiple vendors, and until the company has the root cause, it is not identifying those vendors.”
• Why Android and iPhone won’t go down like BlackBerry – It would be technically impossible for all Android phones or iPhones to experience a global four-day outage like the one BlackBerry saw this week, according to mobile communications experts. Why? The answer is in the technical details of how Research in Motion — the company that makes BlackBerry smartphones, with their click-clacking keyboards and tie-wearing owners — handles e-mails and text messages. From CNN.com.
• Slough: UK town where BlackBerry problems started – BlackBerry’s European hub is flanked by two pharmaceutical companies in near-identical buildings. It sits on a trading estate known to many Britons as the setting for the BBC comedy “The Office”. The series, starring Ricky Gervais as a hapless manager in a paper-making company, mocked the tedium of corporate life in a town known for its many roundabouts and concrete car parks. Slough is the butt of many jokes among Britons. For example, comedian Jimmy Carr said of his hometown: “I grew up in Slough in the 1970s. If you want to know what Slough was like in the 1970s, go there now.” From Reuters.

UPDATE: Blackberry has issued a statement providing some additional details on the outage: “The messaging and browsing delays being experienced by BlackBerry users in Europe, the Middle East, Africa, India, Brazil, Chile and Argentina were caused by a core switch failure within RIM’s infrastructure. Although the system is designed to failover to a back-up switch, the failover did not function as previously tested. As a result, a large backlog of data was generated and we are now working to clear that backlog and restore normal service as quickly as possible. We apologize for any inconvenience and we will continue to keep you informed.”

Monday’s outage  left millions of Blackberry users in the EMEA region (Europe, Middle East and Africa) unable to access their e-mail, instant messages or the Internet. Early this morning Blackberry posted a message to its UK Twitter account saying all Blackberry services had been restored. But within hours, users were complaining that the service was down again. Research in Motion was again slow to acknowledge the new problems, forcing its carrier partners to publicly point the finger.

“We are aware that there are still issues with Blackberry services,” the Orange network tweeted. “RIM are investigating and we will update when we have any info.” Soem partners were more direct. “SEACOM is not responsible for the Blackberry service (BIS) outage currently being experienced,” the South African telecom company tweeted.

The company finally acknowledged the new issues at about via its BlackberryHelp Twitter account. “Some areas have messaging delays and impaired browsing,” it said. “We’re working to restore normal service as quickly as possible.” Late Tuesday the company said the problems had spread to include India, Brazil, Chile, and Argentina.

RIM’s sparse updating stands in marked contrast to the steady stream of complaints from frustrated users, mixed in with links to outage reports and jokes.

The importance of social media in managing customer communication is well established. More than two years ago we noted that “Twitter has emerged as perhaps the most important venue for discussion of hosting outages, powering fast-moving conversations in which unhappy customers share information and complaints. Providers ignore this development at their peril.”

“Our sincere apologies to all of you affected by today’s BlackBerry service issues,” RIM tweeted at 3:45 pm Eastern. “Some users in EMEA are experiencing issues.”

There were reports in several media outlets that the problems in the EMEA region (Europe, Middle East and Africa) had been traced to a problem in a RIM data center in Slough in the UK.

Perhaps not quite yet. Outages at Amazon Web Services’ cloud this spring and summer caused outages at social media sites like HootSuite and Reddit (see DCK coverage in April 2011 and August 2011).  Such cloud outages lead the layman to ask, “How trustworthy is this cloud technology?”

Then last month, retailer Target saw its self-hosted online store knocked offline by a surge in visitors for a widely-advertised sale of limited edition products from the Italian designer Missoni. The downtime occurred three weeks after Target moved off the Amazon cloud, demonstrating that internal infrastructure doesn’t always meet heavy demand, either.

Hybrid – Combinations Give Best of Both Clouds

What’s the solution? Hybrid clouds, cloud bursting and capacity planning, some say. “There is a lot of interest in hybrid clouds,” said Mike Crandell, CEO of Santa Barbara, Calif.-based RightScale, which offers cloud management tools that help IT teams work with multiple cloud infrastructures, deployed across multiple services or locations. The RightScale Cloud Management Platform is delivered as “software as a service” (SaaS) and clients include PBS, Harvard University, Zynga and Sling Media.

“There are advantages to each approach, public or private,” said Crandell. “In a public cloud, the company doesn’t have to invest in infrastructure, hardware. In a private cloud, for some companies making the capital investment (in hardware and infrastructure) is more economical. They can amortize the cost over time. The internal cloud also is good when there is predictable utilization, and for when you need to use it for security and compliance reasons.”

Hybrid clouds can leverage the best of both worlds and allow for cloud bursting -the use of public cloud capacity during times of peak demand. “The biggest example of hybrid cloud is Zynga (social gaming company) which they manage with RightScale,” said Crandell. “They have 250-million-plus users and use a public cloud when a new game launches and it is unpredictable whether the game will go viral. Then when the traffic is more stable, they move their application to an internal cloud.”

“Target (outage) is an old story that has been happening for a long time. Cloud architecture offers a way to expand and meet the load during high demand,” Crandell explained. He described other retailers that worked with RightScale tools, after outages occurred or to prevent them from happening.

For example, the Mars candy company had a coupon for a free chocolate giveaway that brought down its site.RightScale helped the company manage its cloud capacity, and the rest of the 20-week promotion was successful,Crandell said. Mattel Toys had an online social site that it was launching to complement the “American Girl” doll product line. After testing, the company realized its infrastructure would not handle the anticipated load, so they came to RightScale to scale its cloud infrastructure.

Expanding Clouds and Marketplace

Crandell also said RightScale has experienced the growth of the cloud in the number of servers it manages. In September, RightScale hit three million virtual servers under its management. (See RightScale blog post.) Founded in 2007, it took more than two years (27 months) to get to the 1 million server milestone, one more year to reach 2 million, and then half that time to breach 3 million milestone.

Crandell sees continued growth in the cloud marketplace. “It will become more crowded before becoming less so,” he said. “We are seeing some consolidation with the purchases of Terremark, Savvis and CloudSwitch. At the same time, more players are getting into the game. The cloud infrastructures we support are growing.

“It’s almost like rock music in 1965. There were the Beatles, Stones and Dylan, there was clearly a movement. But looking back, we know there was so much more ahead. I still think it’s early days for cloud.”

Cloud leaders include Amazon Web Services and Rackspace, but HP, Dell and Microsoft Azure are also getting into the game. RightScale is partnering with Rackspace on private cloud tools that allow users to buy, provision and set up clouds in 5  to 10 minutes.

RightScale’s tools provide automation, multi-cloud support and governance. “It was built for the cloud as a true-cloud offering, not ‘cloud washed,’ ” said Crandell. “We offer flexibility so companies have a choice of what they run and where they want to run it.”

Upcoming for RightScale is its user conference on November 8-9 in San Jose. “We focus on customers stories – what they’ve built, what have been their challenges and successes.” For more info on the RightScale conference, visit the company’s website.

The traffic burst appears to have been very focused, as data from Akamai doesn’t show any noticeable jump in traffic to news sites as the Apple event began.

“All of our data centers are up, and on generator power,” said Tim Caulfield, CEO of American Internet Services (AIS), which operates three data centers in San Diego. “There have been no issues, and no impact on customer services.”

Reached Thursday night, Caulfield said the company was preparing for an outage lasting 24 to 48 hours.

“Our biggest concern now is a multi-day event where we would need to refuel,” said Caulfield. “Obviously, (fuel deliveries) are being diverted to critical infrastructure like hospitals. We’re confident in our fuel deliveries, but you’re never totally comfortable until the truck is outside the building. We’re preparing for a long event.”

Power was restored early Friday for 1.4 million customers of San Diego Electric & Gas. The outage, which began Thursday afternoon, was triggered when a 500-kilovolt high-voltage line from Arizona to California went out, according to SignOn San Diego. That outage then knocked the San Onofre nuclear power plant offline. Those are the two major power sources for the region.

“Essentially all lines were severed into San Diego County,” said San Diego Gas and Electricity President Michael Niggli , who said about 3 million people in the county were affected by the massive power outage.

ABC News reported that the outage was triggered when the North Gila-Hassayampa 500 kV transmission line near Yuma, Ariz. was tripped offline when a single Arizona Power Service employee was carrying out a procedure in the North Gila substation. Typically, in such an instance, the outage would be isolated to the Yuma area. The investigation is now focusing on the reason that did not occur in this case, APS said Thursday.

The equipment failure resulted in a power surge that caused a “hard shutdown” of the state’s data center, according to The Seattle Times.  Joanne Todd, a spokeswoman for the Department of Information Services,  said generators initially kicked on to maintain the system but that a second surge forced the generators to shut down. Todd said the generators are designed to shut down in that situation to avoid damage to equipment.

The incident occurred on a Sunday afternoon, and the data center was back online by early Monday. The downtime delayed the processing of unemployment claims, but checks went out as scheduled.

The Department of General Administration says the outage will cost the state $400,000 to $500,00 to replace the electrical switch, which was custom-made, and hire an engineering firm to analyze the incident and improve safety and redundancy.

“Equipment in the underground vault remains a life-safety issue and there may be additional changes needed in another vault,” the GA said in a fact sheet, according to The Olympian. “GA has locked the vault and will bar entry to anyone until the equipment is replaced. The replacement work is not expected for about 16 to 20 weeks since the switch must be custom made. The repair will require a weekend power outage.”

The incident occurred at the existing data center for the state, which is also building a new data center in Olympia.