Quake-Proofing An Entire Data Center

The 365 Main flagship data center in San Francisco.

How do you engineer a data center for high availability in an earthquake zone? That’s a special challenge in San Francisco, which is home to a vibrant community of Internet businesses but experienced widespread devastation from a 1906 quake and sustained another big hit in the 1989 Loma Prieta (”World Series”) earthquake.

365 Main houses mission-critical equipment for more than 200 companies in its data center nestled alongside the Bay Bridge. When original owner AboveNet was converting the former tank turret factory for data center use in 2000, it confronted the earthquake risk, installing a base isolation designed to keep the entire building stable when the earth moves.

“This is one of the safest buildings in San Francisco,” said Miles Kelly, senior VP of marketing for 365 Main.

Many data center companies use rack-level earthquake isolation units, which are installed under racks and cabinets and employ a ball-and-cone system to allow the equipment to gently roll back and forth during an earthquake. Providing earthquake protection at the building level involves similar concepts, but a lot more engineering.

Power Savings Free Up Space at 365 Main

The 365 Main flagship data center in San Francisco.

Colocation provider 365 Main says it has space available in its flagship data center in San Francisco for the first time since late 2006. The new capacity is a result of several large customers “graduating” to the company’s new facility across the bay in Oakland, and energy savings created by server consolidations by several customers.

San Francisco has historically been one of the strongest markets for colocation space, according to 365 Main VP of marketing Miles Kelly. “We’re in deep conversations with existing customers to take some of that capacity,” said Kelly. “We expect it to go quickly. San Francisco is a tight data center market.”

Two large customers have relocated to 365 Main’s Oakland data center, which opened in early 2007 and has more than 80,000 square feet of raised-floor technical space. Kelly wouldn’t say which customers, but previous announcements identify blog hosting provider Six Apart and social network Hi5 as customers that have migrated from San Francisco to Oakland.

hi5 Expands at 365 Main Oakland Facility

Fast-growing social network hi5 will expand its data center footprint by taking space in 365 Main’s new facility in Oakland, Calif., the two companies said today. San Francisco-based hi5 has been a tenant at 365 Main’s flagship San Francisco data center since 2004. The expansion is the latest indicator of the growing importance of social networks as major users of data center space, and follows major infrastructure expansions by MySpace and Facebook.

hi5 was founded in 2003 and is now the 8th most-trafficked website in the world according to Alexa. The company will lease an additional 2,500 square feet of space at the Oakland data center, which opened in September 2007.

365 Main joins EPA’s Energy Star

365 Main announced today that it has joined the Energy Star program as a partner, becoming the first data center specialist to participate in the Environmental Protection Agency’s program to rate commercial buildings. 365 Main will also participate in the EPA’s future development of an Energy Star energy performance rating customized for data center facilities.

The specialized nature of data centers makes it more difficult to measure their energy efficiency using programs designed for office buildings. These include the LEED “green building” certification system from the U.S. Green Building Council (USGBC) as well as the Energy Star program for buildings. While it is best known for its energy ratings for appliances, Energy Star has also has certified more 4,350 buildings as meeting its guidelines.

As an Energy Star partner, 365 Main agrees to measure and track energy performance using tools offered by Energy Star, and implement a plan to achieve energy savings consistent with the program’s guidelines.

365 Main Nets Green Savings in Oakland

Energy efficiency efforts at 365 Main’s new Oakland, Calif. data center will reduce annual energy consumption by more than 1.67 million kilowatt hours (kWh), enough energy to power approximately 242 homes each year, the company said today. The effort has earned the 365 Main more than $168,000 in rebates from Pacific Gas & Electric Company (PG&E), the local power utility, to help offset the cost of the improvements.

The measures were implemented through Savings By Design, an energy conservation initiative backed by four of California’s power companies: PG&E, San Diego Gas and Electric, Southern California Edison, and the Sacramento Municipal Utility District (SMUD). “Energy-efficient development is a top priority for us not only because it helps keep energy costs down, but also because we know we’re ultimately reducing our company’s carbon footprint,” said Chris Dolan, CEO of 365 Main.

Customers: Uptime Trumps Green

Ted Samson at InfoWorld’s Sustainable IT reports an interesting wrinkle in 365 Main’s development of a LEED-certified data center in Newark, Calif. 365 Main put considerable effort into a plan to power the facility using generators running on natural gas. The company estimated the gensets would cost $25 million, pay for themselves over time, and save 20,500 tons of carbon per year. 365 Main planned to use the generators as the primary power source, with the local electric grid providing backup power.

There was only one problem: customers rejected the idea, even though it would have meant no additional cost. The reason? By the company’s calculations, the natural-gas-powered generators alone could deliver only a 94 percent guaranteed uptime. Although grid backup would bring the uptime projection to 99.8 percent, customers were unwilling to support a greener solution if it had even a fractional slippage on uptime guarantees.

Six Apart Expands at New 365 Main Site

Data center developer and operator 365 Main today opened its new 111,000-square-foot data center in Oakland, California, which will provide expansion space for Bay Area tenants housed in the company’s founding data center in San Francisco, including blogging pioneer Six Apart. The Oakland facility adds about 80,000 square feet of raised-floor space to 365 Main’s footprint.

365 Main bought the Oakland facility in January and leased one-third of its real estate within the following 90 days. “While we’ve enjoyed tremendous leasing activity at the Oakland building since day one, we’ve made significant upgrades to meet our strict standards for power, cooling, connectivity and security,” said Chris Dolan, 365 Main CEO. “The building isn’t entirely finished, but demand is so high that the decision to officially open our doors for business was an easy one.”

365 Main Offers Apology, Final Report

365 Main has apologized to customers affected by last Tuesday’s power outage at its San Francisco facility, and issued a final incident report. “As I reflect on the last week, I’d like to begin with an extension of our sincere apologies to our San Francisco customers who were impacted by the power incident on July 24th 2007,” President and CEO Chris Dolan said. “Because we strive each day to deliver our customers the world’s finest data centers, we are taking this event very seriously.”

365 Main’s final report confirmed preliminary findings that the outage was caused by flaws in a component known as a Detroit Diesel Electronic Controller (DDEC), a system which monitors a diesel engine’s status, and can activate alarms or shut down a generator. Erroneous data from the DDEC caused 365 Main’s diesel engines to malfunction, failing to start properly after a grid outage.

“In the days since the incident occurred, we identified and corrected the root source of the problem and are taking steps to prevent this type of problem from happening again,” said Dolan. “We are also making our comprehensive findings available to other data centers to try to prevent the same problem from recurring elsewhere.”

365 Main IDs Faulty Generator Parts

365 Main has identified faulty components in the systems that regulate the speed of its generators as they start, the company said Sunday in an update on its investigation into the power outage. In Tuesday’s outage, which knocked many popular sites offline, four of the facility’s 10 diesel generators failed to start properly.

“The digital controller for the diesel engine (know as a DDEC) has proven erratic and a spare DDEC is en route,” 365 Main said, adding that its engineers have been able to reproduce failures in the controllers in testing. “While this component is the focus of the investigation, the team continues start/stop testing to rule out other potential contributors to failure.” DDEC is an acronym for Detroit Diesel Electronic Controller, a system which monitors a diesel engine’s status, and can activate alarms or shut down a generator.

It’s clear that the outage is also becoming a priority for Hitec Power Protection, which makes the generators and flywheel UPS systems used by 365 Main. Staffers from Hitec’s US operation have been on-site in San Francisco since shortly after the outage. On Saturday, Hitec flew in a senior engineer from its headquarters in the Netherlands to join the investigation, along with a member of the company’s board of directors. 365 Main noted Friday that “once a root cause is discovered, we will be introducing a tested fix across all facilities that feature Hitec generators.” 365 Main has seven data centers across the country.