Tenants at the Garland Building (1200 W. 7th) in Los Angeles have been enduring a difficult week. On July 24 the building lost all power for the second time in less than a year, as an automatic transfer switch and UPS backup system failed. The resulting outage made headlines, as it knocked MySpace off the Web for an extended period. Then on July 28, the Garland building suffered *another* loss of power, which was attributed to a “dead short” that occurred while the building was on generator power.
The outages have been especially difficult for Dreamhost, a growing hosting provider which colocates servers in two facilities within Garland (Switch & Data and Alchemy). Dreamhost had been struggling with network problems when the July 24 power outage hit, and has had rolling performance problems ever since. The Dreamhost team maintains a weblog, and today it provided its frustrated customers with a lengthy chapter-and-verse timetable of the problems, titled Anatomy on a(n Ongoing) Disaster.
As Dreamhost’s Josh Jones explains, the repeated outages at 1200 W. 7th are related to power capacity issues and the cost of redundancy.
Here’s an excerpt from Jones’ lengthy post:
Around last June the building informed all its data center tenants that they had essentially run out of power! Not power altogether, but the “good” power that data centers need.. i.e. ups and generator-backed power. Because Wells Fargo, who holds the master lease on the building, wasn’t sure if they were going to renew the lease when it is up in three years, they didn’t want to invest the millions of dollars to add more generators and ups to increase capacity.
In September 2005, much of LA lost grid power when a utility worker accidentally cut a key cable. The Garland building has five generators, but three malfunctioned, and the remaining two were unable to handle the load. In the July 24 outage this year, the generators worked, only to have the ATS and UPS systems fail.
Dreamhost is making the best of matters with a full explanation of the problems, and trying to use humor to dull the pain (interspersing photos of famous disasters throughout the timeline of incidents). While some customers are posting supportive comments, many others are critical of the impact on their business. Several noted that several other builduing tenants, such as MediaTemple and MySpace, had less persistent problems than Dreamhost.