The Downtime Channel is brought to you by ComRent

• Performance Problems for Rackspace Cloud

  January 14th, 2010 : Rich Miller

  Rackspace reports that its cloud computing service is “degraded,” with many customers reporting their sites are unreachable. The company attributed the problem to an unusual load spike in the storage system supporting its cloud platform. The outage came several hours after the Rackspace Cloud disabled CRON, a command commonly used to automate tasks on Unix and Linux systems. By early evening, the company said performance had improved.

  “Starting yesterday we began experiencing very high loads on our storage devices for cluster WC1 in DFW,” Rackspace said on its status page. ”In order to reduce load we have shut down processes like CRON to ensure core site content continue to load.  While load spikes are common in our cloud infrastructure, we have not been able to fully identify the root cause of these unusual issues.

  “We are working with engineers from inside and outside the company with the best expertise on these issues to resolve them and develop a plan of action to ensure we do not repeat this state. We have a series of changes that are being implemented in real time.  We are being careful to minimize issues as we proceed.”

  UPDATE: At about 6 pm Central time Rackspace provided an update: “We have been seeing improved performance on our Cloud Sites WC1 storage cluster for the last few hours. Assuming stability continues we will resume CRON operations this evening. At this time, we cannot declare victory on this issue, but we have many plans in place to continue to increase headroom and ensure stability under all conditions.”

• 
• 
• Slashdot It!

Don

Posted January 14th, 2010

These issues at RackspaceCloud are killing me. Today, was the delivery date of an important nat’l client. Well, I will see if they trust my firm to handle their web affairs after multiple days of nothing. I can’t tweak or show anything and their staff meeting is today. Crap!!!

Dan Landrum

Posted January 14th, 2010

We’re a very small, niche business that chooses to spend more for a service like RackSpace so that we can deliver a fast website to our clients with nearly zero downtime. These outages are killing us too. It isn’t just the downtime, it is what it does to our customer base, like Don said, it is hard enough to establish trust. We then have to expend scarce time explaining what happened to our users.

To add insult to injury, did anyone see the email from RackSpace today announcing a $50 monthly increase, and the reasons why? It said in part.
“We felt it was necessary to update prices for new customers to reflect the value provided from the service, while still remaining very competitive versus other cloud platform as a service offerings.”

jwb

Posted January 15th, 2010

Wow, I really, really, really don’t want my cloud provider reaching into my servers and shutting off my processes. Their intervention should end at gracefully shutting off my instances, not monkeying around with their operating systems.

Guy Hammer

Posted January 15th, 2010

Rackspace has likely oversubscribed the platform — and if they are running CRON jobs on the platform, then it is likely based on the Xen hypervisor, which has fewer high-availability features compared to VMware ESX. I am familiar with these technologies because my firm builds private VM solutions for clients who require a higher level of security and performance. Although ESX is more expensive, we chose it over Xen because of the the enhanced high-availability capabilities.

Anonymous

Posted January 15th, 2010

RS Cloud Sites does not use virtual machines.

When you have thousands of poorly written cron jobs the storage will be affected and it’s the most non impacting way of lowering the load on the storage.

RS Cloud Servers are not being affected by Cloud Site issues. Noone logs into your Cloud Server except for you.

Jane

Posted February 11th, 2010

If you actually want a fast website for your clients with nearly zero downtime use a company that has enterprise solutions like Carpthia Hosting. Rackspace has lost all my trust.

RESOURCE LINKS: