Network Issue Cited in Rackspace Outage

Rackspace says a network peering problem caused an outage this afternoon that affected its Cloud Sites cloud computing service. The incident resulted in downtime for some sites hosted in the company’s Dallas data center, which has experienced several outages this year due to power problems. But Rackspace said the problems originated outside the Dallas facility.

Rackspace said the incident began at 3:42 pm and the network was restored at 4:13 p.m. Discussion on networking groups suggested Rackspace may have experienced a “routing loop” in which packets continue to be routed in an endless circle, which can result from hardware failures or configuration problems.

UPDATE: “The issues resulted from a problem with a router used for peering and backbone connectivity located outside the data center at a peering facility, which handles approximately 20% of Rackspace’s Dallas traffic,” Rackspace said in an incident report on its blog. “The problems stemmed from a configuration and testing procedure made at our new Chicago data center, creating a routing loop between the Chicago and Dallas data centers. This activity was in final preparation for network integration between the Chicago and Dallas data centers. The network integration of the facilities was scheduled to take place during the monthly maintenance window outside normal business hours, and today’s incident occurred during final preparations.”

Rackspace and the Transition to the Cloud

John Engates jokes that web hosting is “the original cloud computing.” It’s a reminder of the connection between the past and future for Rackspace Hosting. As the chief technical officer of Rackspace, Engates has helped steer the company through transitions driven by both technology and terminology.

As customers focus on the benefits of cloud computing, hosting companies are pondering the best way to reposition their product offerings to compete. Perhaps no hosting provider has navigated the shift to the cloud more smoothly than Rackspace (RAX), a company founded in the early days of the dot-com boom that has emerged as the second-largest player in cloud computing, according to some analyses.

In the first three quarters of 2009, Rackspace’s cloud computing operation grew from 34,820 customers to 61,616. The company’s revenue from cloud computing in the third quarter was $15.3 million, a fraction of the $147 million brought in by the company’s managed hosting business, but more than double the $6.5 million from the same period in 2008. 

Build, Buy or Partner
In its journey to the cloud, Rackspace confronted a key question facing hosting providers entering the cloud computing market: do we build, buy or partner? For Rackspace the answer has been “all of the above.” Engates says this approach has provided Rackspace with the flexibility to address a variety of customer decisions.

Rackspace developed in-house capabilities through its Mosso unit, which started life as a reseller offering and evolved into the Rackspace Cloud. The San Antonio company also acquired e-mail specialist Webmail.us in 2007 and cloud storage specialist JungleDisk and virtual private server provier SliceHost in 2008. It also added capabilities through partnerships, such as a deal with Limelight Networks to offer content delivery to customers.  

“We got an early start compared to some of our competitors,” Engates says of Rackspace’s push into cloud computing. “Many of our hosting competitors wanted to wait and see how things developed. Mosso gave us the engineering capability and confidence to go do it. It helped us do a reboot.”

Roundup: Ciena, Equinix, Rackspace

Here’s a roundup of news announcements from the data center and hosting industry:

• Ciena acquires Nortel Ethernet assets. Ciena Corporation announced that they were selected as the successful bidder in the auction for all of the optical networking and carrier Ethernet assets of Nortel’s Metro Ethernet Networks business.  Ciena has agreed to pay $769 million for the assets.  Ciena CEO Gary Smith said the purchase is “bringing together complementary technologies in switching and transport to create an innovative powerhouse with the scale to challenge the industry status quo and offer customers a practical path for transitioning to automated, optical Ethernet-based networking.”  In 2008 Nortel saw $1.36 billion in revenue from the assets to be acquired.
• Equinix adds to European Ethernet Exchange.  Earlier in the month Equinix announced that submarine transport cable provider Hibernia Atlantic was expanding with Equinix in New York. Now European-based carriers Exponential-e and Tinet have signed up to participate in the recently announced Carrier Ethernet Exchange platform.  The initial deployment locations include London, New York, Chicago and Silicon Valley.  The goal of the project is to provide Ethernet Network to Network Interconnections (E-NNI).  VP of Engineering at Exponential-e Mukesh Bavisi said “Equinix already plays a strategic role in our peering relationships with other Tier 1 service providers, so it is a natural choice that we leverage its carrier-rich interconnection hubs to develop an Ethernet interconnection infrastructure.”  According to Infonetics Research demand for carrier Ethernet services is set to double within five years.
• Rackspace offers complete Verisign SSL line.  Rackspace announced that they expanded an agreement with Verisign allowing them to directly sell, install and renew Verisign SLL Certificates to their customers.  Additionally the entire line of certificates is now available as a self-service option via the MyRackspacecom customer portal.  “Trust is the most valuable currency on the Internet today, and trust is what VeriSign SSL and EV SSL protection delivers for Web site operators the world over,” said Michael Lin, vice president and general manager of SSL at VeriSign.

Roundup: Internap, Rackspace, Joyent, Isilon

Here’s a roundup of news announcements from the data center and hosting industry:

• Internap Enhances CDN for High Quality Video. On Tuesday Internap Network Services announced enhancements to its content delivery network (CDN) offering, including new ease-of-use functions and automation of key CDN capabilities.  A comScore Video Metrix service poll showed that in September 2009 more than 84.8% of the total U.S. Internet audience watched online video. Version 5.0 of Internap’s CDN MediaConsole includes an integrated rule-based transcoding that automatically converts video into the ideal formats for a broad range of devices.  It also performs a continuous bitrate adjustment that dynamically adapts video streams based on the capabilities and bandwidth of the network.  Internap is showcasing its CDN offering at the Streaming Media West conference November 17-19.
• Rackspace Launches Cloud Drive. Rackspace (RAX) announced a set of new product offerings that help businesses move their IT applications into the cloud.  New products include Cloud Drive, Server Backup, and an upcoming release of Hosted Microsoft SharePoint.  The new cloud products mark an aggressive move by Rackspace to expand into collaboration and backup applications and compete in the software-as-a-service market.  Rackspace Cloud Drive is a cloud-based, online file storage application that allows individuals and teams to store, share and backup files. Rackspace Server Backup is a cloud-based, online server backup application designed to protect file server data.  Both Server Backup and Cloud Drive are powered by technology from Jungle Disk, a Rackspace subsidiary acquired in 2008.

Rackspace Expands Its Headquarters

Today Rackspace Hosting (RAX) will open the 120,000 square foot second phase of its headquarters facility within a former shopping mall in San Antonio. Last week we visited Rackspace’s digs inside the 1.1 million square foot facility, which showcases the company’s corporate culture and its emphasis on “Fanatical Support.” The expansion space will soon be the home of the company’s fast-growing cloud computing operation, which is among the departments being consolidated from the company’s other office sites in the San Antonia area. And yes, they have a Big Ass Fan. This video runs about 3 minutes, 20 seconds.

For more information about Rackspace Hosting, check out our Rackspace Channel. For additional video, check out our DCK video archive and the Data Center Videos channel on YouTube.

Posterous Moves Storage to Rackspace Cloud

It’s been a busy day for Rackspace Hosting (RAX). The company announced a high-profile storage customer win, while two Wall Street analysts issued buy recommendations on Rackspace and raised their price projections for the stock.

Rackspace said today that hot insta-blogging service Posterous is shifting its storage from Amazon S3 to Rackspace’s Cloud Files’ service. Posterous had already been using Rackspace-owned SliceHost for its cloud compute needs, and decided to shift its storage as well. Posterous allows users to create blogs in minutes and update their page with text, pictures or video sent directly to an email address.

“We built a dead simple application for everyone – from geeks to soccer moms – to share thoughts, pictures and videos, and we needed a cost effective hosting solution to run it,” said Sachin Agarwal, Co-Founder of Posterous. “We had turned to The Rackspace Cloud for affordability, the ability to spin up new servers in seconds, uptime and for Fanatical Support. We’ve been so happy with the service that we have now made the decision to switch to Rackspace Cloud Files, from Amazon’s S3, for our new storage needs.”

Rackspace: ‘We Have Work To Do’

Rackspace has apologized to customers of its cloud computing service for an outage early Tuesday, and provided additional details about the incident. The company said it still has “high confidence” in the capabilities of its Dallas/Fort Worth data center, which has been plagued by a series of power events this year that have disrupted customer service.

“The experience you had last night is not acceptable to us,” wrote Emil Sayegh, General Manager of The Rackspace Cloud, in an e-mail to customers. “We have work to do to earn back your trust. We will not rest until we have.” As part of that process, Rackspace is reviewing its procedures for recovering from outages in its cloud computing operation.

Power Loss at PDUs
The downtime occurred during maintenance as Rackspace continued to work on refinements to the power infrastructure at the DFW data center. A short during testing of a Power Distribution Unit (PDU) caused a loss of power to all PDUs in Cluster G of the Dallas facility.

Power Outage Affects Rackspace Cloud

Rackspace reports that parts of its Dallas data center lost power early today during testing of power distribution units (PDUs) during scheduled maintenance. This resulted in downtime for sites hosted on SliceHost and The Rackspace Cloud, including the leading tech blog TechCrunch, which ensured that the outage was widely noted on blogs and Twitter.

The Dallas data center has experienced power problems before, including outages on June 29 and July 7 that prompted Rackspace CEO Lanham Napier to issue an apology to customers and provide a detailed explanation of the outage and the operations of the Dallas/Fort Worth facility. 

Rackspace Adds Storage, DR Offerings

In a reflection of the growing importance of large-scale storage and backup, Rackspace Hosting (RAX) today introduced new managed storage and disaster recovery services targeted for enterprise customers. The San Antonio managed hosting specialist said the new offerings are a response to customers wanting to increase their online storage.

“We evolve our enterprise offerings based on customer,” said Taylor Rhodes, GM of Enterprise Services for Rackspace. “They’re telling us they need a partner to provide important but non-strategic functions, and they want to buy them as services rather than pouring their own capital and IT resources into them. We are now able to offer our most complex customers solutions designed for their storage and data replication needs.”