Rackspace: ‘We Have Work To Do’

Rackspace has apologized to customers of its cloud computing service for an outage early Tuesday, and provided additional details about the incident. The company said it still has “high confidence” in the capabilities of its Dallas/Fort Worth data center, which has been plagued by a series of power events this year that have disrupted customer service.

“The experience you had last night is not acceptable to us,” wrote Emil Sayegh, General Manager of The Rackspace Cloud, in an e-mail to customers. “We have work to do to earn back your trust. We will not rest until we have.” As part of that process, Rackspace is reviewing its procedures for recovering from outages in its cloud computing operation.

Power Loss at PDUs
The downtime occurred during maintenance as Rackspace continued to work on refinements to the power infrastructure at the DFW data center. A short during testing of a Power Distribution Unit (PDU) caused a loss of power to all PDUs in Cluster G of the Dallas facility.

“The power disruption was approximately 5 minutes in duration,” Sayegh wrote. “Despite this short power disruption, many customers experienced downtime that was significantly longer. Since the power disruption hit the core of many of our cloud services, recovery of full operations required more effort than simple recovery of power.”

Hard Reboot, Unforeseen Failures
Sayegh said that athough the power outage was brief, it forced a hard reboot on some of Rackspace’s cloud infrastructure. “As our engineers worked to bring hardware back online, we experienced several unforeseen hardware failures,” Sayegh added. “Further complicating our recovery effort, the incident also created internal DNS issues, which caused additional delays. With that said, the vast majority of cloud customers affected by this outage had service restored within one hour’s time … however, depending upon the service, a few customers experienced service interruptions for up to few hours.”

The outage comes after Rackspace vowed a thorough review of the Dallas facility, which experienced power-related outages on June 29 and July 7, as well as a significant outage in 2007. Rackspace CEO Lanham Napier said the company would “bring every resource to bear” on improving the reliability of the DFW facility.

Sayegh said Rackspace has “invested massively in the DFW facility to ensure it delivers at a level you expect from Rackspace – despite last night, we feel very good about our plan and have high confidence in the DFW facility – clearly we have to prove it.”