hi5 Expands at 365 Main Oakland Facility

Fast-growing social network hi5 will expand its data center footprint by taking space in 365 Main’s new facility in Oakland, Calif., the two companies said today. San Francisco-based hi5 has been a tenant at 365 Main’s flagship San Francisco data center since 2004. The expansion is the latest indicator of the growing importance of social networks as major users of data center space, and follows major infrastructure expansions by MySpace and Facebook.

hi5 was founded in 2003 and is now the 8th most-trafficked website in the world according to Alexa. The company will lease an additional 2,500 square feet of space at the Oakland data center, which opened in September 2007.

365 Main joins EPA’s Energy Star

365 Main announced today that it has joined the Energy Star program as a partner, becoming the first data center specialist to participate in the Environmental Protection Agency’s program to rate commercial buildings. 365 Main will also participate in the EPA’s future development of an Energy Star energy performance rating customized for data center facilities.

The specialized nature of data centers makes it more difficult to measure their energy efficiency using programs designed for office buildings. These include the LEED “green building” certification system from the U.S. Green Building Council (USGBC) as well as the Energy Star program for buildings. While it is best known for its energy ratings for appliances, Energy Star has also has certified more 4,350 buildings as meeting its guidelines.

As an Energy Star partner, 365 Main agrees to measure and track energy performance using tools offered by Energy Star, and implement a plan to achieve energy savings consistent with the program’s guidelines.

365 Main Nets Green Savings in Oakland

Energy efficiency efforts at 365 Main’s new Oakland, Calif. data center will reduce annual energy consumption by more than 1.67 million kilowatt hours (kWh), enough energy to power approximately 242 homes each year, the company said today. The effort has earned the 365 Main more than $168,000 in rebates from Pacific Gas & Electric Company (PG&E), the local power utility, to help offset the cost of the improvements.

The measures were implemented through Savings By Design, an energy conservation initiative backed by four of California’s power companies: PG&E, San Diego Gas and Electric, Southern California Edison, and the Sacramento Municipal Utility District (SMUD). “Energy-efficient development is a top priority for us not only because it helps keep energy costs down, but also because we know we’re ultimately reducing our company’s carbon footprint,” said Chris Dolan, CEO of 365 Main.

Customers: Uptime Trumps Green

Ted Samson at InfoWorld’s Sustainable IT reports an interesting wrinkle in 365 Main’s development of a LEED-certified data center in Newark, Calif. 365 Main put considerable effort into a plan to power the facility using generators running on natural gas. The company estimated the gensets would cost $25 million, pay for themselves over time, and save 20,500 tons of carbon per year. 365 Main planned to use the generators as the primary power source, with the local electric grid providing backup power.

There was only one problem: customers rejected the idea, even though it would have meant no additional cost. The reason? By the company’s calculations, the natural-gas-powered generators alone could deliver only a 94 percent guaranteed uptime. Although grid backup would bring the uptime projection to 99.8 percent, customers were unwilling to support a greener solution if it had even a fractional slippage on uptime guarantees.

Six Apart Expands at New 365 Main Site

Data center developer and operator 365 Main today opened its new 111,000-square-foot data center in Oakland, California, which will provide expansion space for Bay Area tenants housed in the company’s founding data center in San Francisco, including blogging pioneer Six Apart. The Oakland facility adds about 80,000 square feet of raised-floor space to 365 Main’s footprint.

365 Main bought the Oakland facility in January and leased one-third of its real estate within the following 90 days. “While we’ve enjoyed tremendous leasing activity at the Oakland building since day one, we’ve made significant upgrades to meet our strict standards for power, cooling, connectivity and security,” said Chris Dolan, 365 Main CEO. “The building isn’t entirely finished, but demand is so high that the decision to officially open our doors for business was an easy one.”

365 Main Offers Apology, Final Report

365 Main has apologized to customers affected by last Tuesday’s power outage at its San Francisco facility, and issued a final incident report. “As I reflect on the last week, I’d like to begin with an extension of our sincere apologies to our San Francisco customers who were impacted by the power incident on July 24th 2007,” President and CEO Chris Dolan said. “Because we strive each day to deliver our customers the world’s finest data centers, we are taking this event very seriously.”

365 Main’s final report confirmed preliminary findings that the outage was caused by flaws in a component known as a Detroit Diesel Electronic Controller (DDEC), a system which monitors a diesel engine’s status, and can activate alarms or shut down a generator. Erroneous data from the DDEC caused 365 Main’s diesel engines to malfunction, failing to start properly after a grid outage.

“In the days since the incident occurred, we identified and corrected the root source of the problem and are taking steps to prevent this type of problem from happening again,” said Dolan. “We are also making our comprehensive findings available to other data centers to try to prevent the same problem from recurring elsewhere.”

365 Main IDs Faulty Generator Parts

365 Main has identified faulty components in the systems that regulate the speed of its generators as they start, the company said Sunday in an update on its investigation into the power outage. In Tuesday’s outage, which knocked many popular sites offline, four of the facility’s 10 diesel generators failed to start properly.

“The digital controller for the diesel engine (know as a DDEC) has proven erratic and a spare DDEC is en route,” 365 Main said, adding that its engineers have been able to reproduce failures in the controllers in testing. “While this component is the focus of the investigation, the team continues start/stop testing to rule out other potential contributors to failure.” DDEC is an acronym for Detroit Diesel Electronic Controller, a system which monitors a diesel engine’s status, and can activate alarms or shut down a generator.

It’s clear that the outage is also becoming a priority for Hitec Power Protection, which makes the generators and flywheel UPS systems used by 365 Main. Staffers from Hitec’s US operation have been on-site in San Francisco since shortly after the outage. On Saturday, Hitec flew in a senior engineer from its headquarters in the Netherlands to join the investigation, along with a member of the company’s board of directors. 365 Main noted Friday that “once a root cause is discovered, we will be introducing a tested fix across all facilities that feature Hitec generators.” 365 Main has seven data centers across the country.

ValleyWag Retracts Report on 365 Main

The editor of ValleyWag has acknowledged that a post blaming a “drunk employee” for this week’s data center outage at 365 Main was inaccurate. “I was wrong,” ValleyWag Editor Owen Thomas wrote Thursday. “I trusted a source I shouldn’t have.” Thomas said the erroneous tip was sent to him over AOL Instant Messenger from a man who said he worked at a “large software company.” Thomas checked to see that an employee by that name worked for the company, and then posted an item on ValleyWag detailing the tipster’s claim that a drunken employee damaged equipment, causing an outage that knocked many popular sites offline. In reality, several generators at 365 Main’s San Francisco data center failed to start when the facility lost grid power.

Thomas said he was a victim of a prank. “I’ve since learned that the tipster has associations with a band of hackers who delight in social engineering, the art of using technical means to get human beings to do their bidding,” Thomas writes. “I’m told by people close to the hacker group that they do this kind of thing for sheer amusement. That they enjoy harassing people in the Web 2.0 world.”

Generator Failures Caused 365 Main Outage

Several generators at 365 Main’s San Francisco data center failed to start when the facility lost grid power Tuesday afternoon, causing an outage that knocked many of the web’s most popular destinations offline for several hours. The disruption, which began at 1:45 pm PST, occurred during a grid outage for Pacific Gas & Electric, which left significant portions of San Francisco in the dark. Parts of 365 Main’s data center lost power, causing downtime for customer sites including CraigsList, Technorati, LiveJournal, TypePad, AdBrite, the 1Up gaming network, Second Life and Yelp, among others.

Wild rumors circulated about why 365 Main’s backup systems failed to maintain power to key systems, including reports of employee sabotage or a possible triggering of the facility’s emergency power off (EPO) button, a frequent cause of outages at mission-critical facilities. While less sensational, the actual cause of the outage was the failure of backup diesel generators.

“An initial investigation has revealed that certain 365 Main back-up generators did not start when the initial power surge hit the building,” the company said in an incident report. “On-site facility engineers responded and manually started affected generators allowing stable power to be restored at approximately 2:34 pm across the entire facility.”

“As a result of the incident, continuous power was interrupted for up to 45 minutes for certain customers,” the report continued. “We’re certain 3 of the 8 colocation rooms were directly affected, and impact on other colocation rooms is still being investigated.”

The 365 Main data center is supported by 10 Hitec 2.1 megawatt generators, which are tested every month. The 277,000 square foot 365 facility is partitioned into eight data center “pods,” some of which remained online while others went dark.

The facility’s backup systems use flywheel UPS systems - rather than batteries - to provide “ride-through” electricity to keep servers online until the diesel generator can start up and begin powering the facility. A flywheel is a spinning cylinder which generates power from kinetic energy, and continues to spin when grid power is interrupted. In most data centers, the UPS (uninterruptible power supply) system draws power from a bank of large batteries. AboveNet, the original builder/owner of the 365 Main data center, was an early adopter of flywheel UPS systems, which have recently gained attention as a “greener” alternative to batteries.

365 Main Commits to LEED Data Centers

Data center developer 365 Main Inc. has made a commitment to full compliance with the LEED “green building” certification system from the U.S. Green Building Council (USGBC), a non-profit organization of leaders from every sector of the building industry.

365 Main said its new data center in Newark, Calif. will be LEED-certified. Design and construction has already begun on the 136,410-square-foot facility, which will open in the fourth quarter of this year 2007.

Only a handful of data centers have been certified under the LEED standard, including the Highmark Data Center in Pennsylvania and the FNMA Technology Center in Urbana, Md. The Los Angeles County Data Center is expected to achieve LEED certification when it is completed later this year, according to local officials.