Generator Failures Caused 365 Main Outage

Several generators at 365 Main’s San Francisco data center failed to start when the facility lost grid power Tuesday afternoon, causing an outage that knocked many of the web’s most popular destinations offline for several hours. The disruption, which began at 1:45 pm PST, occurred during a grid outage for Pacific Gas & Electric, which left significant portions of San Francisco in the dark. Parts of 365 Main’s data center lost power, causing downtime for customer sites including CraigsList, Technorati, LiveJournal, TypePad, AdBrite, the 1Up gaming network, Second Life and Yelp, among others.

Wild rumors circulated about why 365 Main’s backup systems failed to maintain power to key systems, including reports of employee sabotage or a possible triggering of the facility’s emergency power off (EPO) button, a frequent cause of outages at mission-critical facilities. While less sensational, the actual cause of the outage was the failure of backup diesel generators.

“An initial investigation has revealed that certain 365 Main back-up generators did not start when the initial power surge hit the building,” the company said in an incident report. “On-site facility engineers responded and manually started affected generators allowing stable power to be restored at approximately 2:34 pm across the entire facility.”

“As a result of the incident, continuous power was interrupted for up to 45 minutes for certain customers,” the report continued. “We’re certain 3 of the 8 colocation rooms were directly affected, and impact on other colocation rooms is still being investigated.”

The 365 Main data center is supported by 10 Hitec 2.1 megawatt generators, which are tested every month. The 277,000 square foot 365 facility is partitioned into eight data center “pods,” some of which remained online while others went dark.

The facility’s backup systems use flywheel UPS systems - rather than batteries - to provide “ride-through” electricity to keep servers online until the diesel generator can start up and begin powering the facility. A flywheel is a spinning cylinder which generates power from kinetic energy, and continues to spin when grid power is interrupted. In most data centers, the UPS (uninterruptible power supply) system draws power from a bank of large batteries. AboveNet, the original builder/owner of the 365 Main data center, was an early adopter of flywheel UPS systems, which have recently gained attention as a “greener” alternative to batteries.

365 Main Commits to LEED Data Centers

Data center developer 365 Main Inc. has made a commitment to full compliance with the LEED “green building” certification system from the U.S. Green Building Council (USGBC), a non-profit organization of leaders from every sector of the building industry.

365 Main said its new data center in Newark, Calif. will be LEED-certified. Design and construction has already begun on the 136,410-square-foot facility, which will open in the fourth quarter of this year 2007.

Only a handful of data centers have been certified under the LEED standard, including the Highmark Data Center in Pennsylvania and the FNMA Technology Center in Urbana, Md. The Los Angeles County Data Center is expected to achieve LEED certification when it is completed later this year, according to local officials.

Generator Test Saves Power, Money

Is it possible to hug a tree while exercising a 2.1 Megawatt diesel generator? Not precisely. But 365 Main’s flagship San Francisco data center was able to save $70,000 in power costs last year from the local utility, Pacific Gas and Electric (PG&E), by altering the procedures for monthly testing of its backup generators. PG&E highlighted 365 Main’s efforts yesterday as part of a broader initiative to encourage data centers to reduce their energy use and impact upon the environment.

“365 Main is truly walking the walk in the data center segment, and we encourage other companies to follow its lead in order to protect all types of businesses from potential power outages on critical peak days and to preserve the environment,” said Al Steubing, PG&E Account Services Director. 365 Main is a participant in PG&E’s Critical Peak Pricing (CPP) program, designed to reduce stress on the good during periods of heavy power demand.

As part of its maintenance program, 365 Main tests each of its 10 Hitec generators once a month by running each of the 3,000 horsepower diesel engines for two hours. The company developed a new testing procedure that allowed it to reduce the use of load banks, the 8-foot tall heating coils traditionally used to test generators by drawing immense amounts of power. The new procedure reduced the facility’s utility power consumption by as much as 12.5 percent during monthly tests, and is compliant with guidelines set by the Bay Area Air Quality Management District (BAAQMD).

365 Main Unveils Further Expansion

365 Main Inc. continues to expand its network of data centers, adding two facilities in California markets where existing centers are nearing capacity. The San Francisco-based company today announced the acquisition of a facility in Oakland, Calif. and plans to construct a new data center in Vernon, Calif. (Los Angeles market) to meet continuing strong demand for data center space. The announcement comes as the company has sold out its flagship facility in San Francisco and as its El Segundo, Calif. data center, which opened in June 2006, is nearing capacity.

On Tuesday 365 Main closed on the acquisition of an 111,000 square-foot Oakland data center near Jack London Square, which includes 80,000 square feet of technical space with a 30-inch raised floor. Metro PCS Wireless Inc. and T-Mobile USA are existing tenants in the property, which was built in 2001. 365 Main is investing in additional infrastructure for power, cooling, connectivity and security at the property, which is less than 10 miles from 365 Main’s original San Francisco data center. 365 Main says demand for the remaining space in Oakland is significant, and the first lease for a private colocation room has already been signed by GNi, a California-based managed services provider that provides support services for 365 Main tenants and leases space at most of its facilities.

“The demand for high-quality data center space in the San Francisco Bay Area is the greatest in the country, and we’ve simply run out of room at our San Francisco facility,” said Chris Dolan, chief executive officer of 365 Main. “Expanding to Oakland fits all our criteria: existing demand, existing tenants, good location, well-defined pipeline. It’s a strategic move that makes sense for our business, and we look forward to becoming part of the Oakland community.”

365 Main Targets Hosting, Colo Providers

365 Main Inc. today introduced SCALE 365, a national program in which says it will make improved data center real estate available at wholesale rates. The program is designed for companies that need to build or expand large footprints of modern data center space quickly, including hosting and colocation providers looking to expand.

The program offers power, cooling, connectivity and security services in customizable private rooms of 5,000 to 20,000 square feet within 365 Main’s data centers. The company said SCALE 365 “also extends to the development of new, build-to-suit facilities.” For hosting companies, it offers the opportunity to remarket leased space quickly at favorable margins. Inaugural customers of SCALE 365 include Global Netoptex Inc. (GNi), Internap and NET2EZ.

MMO Game Host Deploys 1,000 Blade Servers

Multiplayer online game host Online Game Services Inc. (OGSi) has deployed more than 1,000 IBM BladeCenter servers at 365 Main’s flagship data center in San Francisco, the companies announced today. OGSi, the gaming channel of managed services provider Global Netoptex Inc. (GNi), says it already has more than 1 million users, with clients including (GoPets Ltd.) and Ping0 LLC, which will distribute the online component to Hellgate: London.

OGSi has invested over five years and more than $20 million in its network infrastructure. The servers installed at 365 Main include a mix of IBM LS20 (AMD-based), HS20 and HS21 (Intel-based), and JS20 (PowerPC-based) blades, and represent the single largest deployment of IBM blade servers in the digital entertainment industry, according to OGSi. The company says it expects to have more than 3,500 blade servers operational by the end of 2007.

“Our on-demand ‘pay-as-you-grow’ game hosting business model is based around a dense hosting environment that requires advanced, modern data centers like 365 Main’s,” said James Hursthouse, CEO and owner of OGSi. “We chose 365 Main because it offers world-class power, connectivity, cooling and security, which together allow us to provide our customers with the finest hosting infrastructure available to power their games.”

Online Game Services Inc. launched in June 2004, and is providing online game companies with fully managed dedicated game hosting services, eliminating the need for MMORPG operators to spend millions of dollars on web infrastructure. “It’s a drastic reduction in risk and cost,” said Hursthouse, who said the MMO sector is growing quickly due to the success of World of Warcraft and strong media interest in Second Life. “There definitely has been a paradigm shift. Right now there aren’t many publishers that aren’t looking at MMOs.”

1Up Gaming Network Moves to 365 Main

Ziff Davis Game Group has moved the servers for its 1UP Network of Web properties to 365 Main’s San Francisco data center. Among the sites moving to the new location are 1UP.com, GameVideos.com, MyCheats.com and the Gazerk.com search engine, which searches the web for game-related content. The 1Up Network serves more than 120 million pages a month to its audience.

The Game Group had been hosting the network at a data center in New Jersey and managing it from San Francisco, which created some logistical challenges. “It was difficult to maintain the data center remotely, and it was a bit complicated when it came to accounting for shared bandwidth and storage,” said John Wegis, chief technology officer of Ziff Davis’ 1UP Network. “Plus we wanted to be able to send IT people on-site quickly, which just wasn’t possible in that situation.” The servers are now in a colo environment a few blocks away from Ziff Davis Game Group’s office in downtown San Francisco.

BitTorrent Inc. Continues to Grow at 365 Main

BitTorrent Inc. has expanded its data center presence in 365 Main’s San Francisco facility. After placing a single rack at the facility in February 2005, BitTorrent’s installation at 365 Main grew to three racks earlier this year, and the company has now expanded into a 600-square-foot server cage, roughly 28 racks.

BitTorrent’s peer-assisted content distribution platform enables users worldwide to publish, search and download popular digital content quickly. The company’s data center requirements have grown significantly as it has signed recent media distribution deals with Warner Bros. Home Entertainment and a number of independent movie distributors, which has increased its digital library to thousands of titles.

“BitTorrent is not a company that limits itself by convention,” said Ashwin Navin, president and co-founder of BitTorrent. “With our aggressive growth plans we need a data center that can grow with us and deliver reliable power, cooling and connectivity. 365 Main provides these features within a stone’s throw away from our engineering team in San Francisco.”