An analysis of data center failures shows that focusing reliability efforts on generators is the best way to improve uptime.

When it comes to reliability, diesel generators are far and away the most important pieces of equipment in a data center, and regulatory mandates on fuel may be creating new problems that could raise generator failure rates.

Those were the key points made by Steve Fairfax, President of MTechnology, in a provocative keynote presentation Tuesday at the 7×24 Exchange Fall Conference in Phoenix. Fairfax, whose firm does “science risk” consulting work for both vendors and end users, said in-depth analyses of failure rates in data center components and systems yields counter-intuitive results, especially when it comes to maintenance.

“Generators are the most critical systems in the data center,” said Fairfax, whose studies of failure data found generators played a role in between 45 and 65 percent of outages in data centers with an N+1 configuration (with one spare backup generator).  ”Reducing generator failures has more than 10 times the impact of reducing other component rates. This is where you should be focusing your attention – on generators. That’s what will take you down.”

Fairfax identified three key threats to generator reliability – fuel quality problems due to old fuel mixing with newer fuel, quality issues with new Ultra-Low Sulphur Diesel and biodiesel fuels, and wear and tear from efforts to start cold generators as quickly as possible.

Fuel Tanks and the “Diesel Solera”

Fairfax said the leading problem with generators is not the failure to start, but the failure to run properly once the generator has started. A key  factor in the “failure to run” scenario is fuel quality.

Fairfax highlighted a phenomenon he calls the diesel “solera,” a term for the process for aging wine by mixing small amounts of older vintages with newer wine. While the solera process can help improve wine, it can introduce reliability challenges when it occurs in a tank of diesel fuel – which happens when older fuel remains in the bottom of a tank when it is refilled.

“Every year we take some of the diesel fuel out and add fresh fuel,” said Fairfax. “When was the last time you emptied that tank and cleaned it out?”

Ultra-Low Sulphur Diesel and BioDiesel

Another factor is the regulatory requirement to use Ultra Low-Sulphur Diesel Fuel (ULSD). While ULSD improves the emissions profile of generators, Fairfax said data center operators should pay close attention to fuel quality and tank conditions. Ultra-Low Sulphur Diesel is less stable than older distillate diesel fuels, he said, with a maximum storage time of 6 months. “Stabilizers can extend that, but then you have an interesting chemstry experiment going on in your diesel tank,” said Fairfax, who says this could result in a higher incidence of leaks and accelerated wear on seals.

Fairfax offered three recommendations on managing this challenge:

• Empty and inspect fuel tanks whenever possible.
• Change your generator testing policies. “Test them as you will run them,” said Fairfax, who said tests should run for 24 to 72 hours to simulate an extended utility outage, which will draw down diesel supplies in ways not seen in shorter periods.
• Sample your diesel fuel on a regular basis to track fuel quality.

Biodiesel, which is mandated in several states (including Massachusetts, Washington and New Jersey), poses additional challenges. “Biodiesel has a completely different chemistry” than older diesel fuels, said Fairfax, who said biodiesel can dissolves sediments that could clog filters, and has even worse stability than ULSD.

10-Second Start Times

Fairfax noted the NFPA guide to Emergency Power Supply Systems calls for generators to be able to start in 10 seconds for life safety applications. While not all data centers are required to adhere to this, many follow the NFPA guidelines by default. Farfax said Mtechnology’s research found no basis for the 10-second guidance, which he said places high stress on the generators that can shorten lifespan and impact reliability. The 10-second target requires cold equipment to start with a wide-open throttle, creating the highest possible thermal and mechanical stress.

“You get a huge benefit by reducing stresses. One of the best things you could do to improve the reliability of generators is to increase the start time to 30 seconds.”

“Our two data centers in Tokyo are operating as normal,” said Kei Furuta, managing director, Equinix Japan. “There is no facility damage or operational impact since the earthquake happened. So far, we have not heard any news about earthquake damage or operational impact to any data center in Tokyo.”

While no data centers in Tokyo were damaged, their resiliency may be tested by a series of rolling blackouts being implemented by Tokyo Electric Power Co. (TEPCO) to cope with the loss of generating capacity at damaged nuclear plants in northern Japan. Over the weekend TEPCO announced plans to implement rolling blackouts to many cities in suburban Tokyo for three to six hours daily. The blackouts will require data centers to switch over to backup generators for extended periods of time. Access to diesel fuel to power the generators will be a key issues should the rolling blackouts persist. Equinix said it has arranged contracts that provide priority access to diesel fuel.

“The biggest concern at the moment is power disruption,” said Furuta. “We have fueled the generators at our Tokyo data centers to their full capacity, which will provide emergency backup power in the event of any power disruption. We have a priority contract with our fuel supply company. We are closely monitoring the situation, but depending on future status, they could have to prioritize among the priority contractors, or in an extreme case, the government may have to prioritize the national energy to the devastated area. We do not expect an issue for fuel in the short term, but the mid to long term future is always unknown.”

Significantly, Furuta said the government recognizes the important role played by data centers in keeping critical services operating in the aftermath of the disasters.

“The government (Ministry of Economy, Trade and Industry) is working on prioritizing power and energy supply to data centers which are considered critical information and telecommunication, and we are working with them,” he said.

An aerial view of the Microsoft data center in Quincy, Washington

Economic development officials love clusters of huge Internet data centers. But environmental officials are less enthused about large clusters of diesel generators. The town of Quincy, Washington has both, serving as home to major data centers for Microsoft, Yahoo and Intuit (with another project from Sabey Corp. on the way).  

Each of the data centers in Quincy uses banks of generators to provide backup power in the event of a utility outage. Diesel engine exhaust is a regulated pollutant, and can be toxic in high concentrations, so state regulators routinely review permits for generators. 

Using 48 Megawatts Now
Earlier this year Microsoft began construction on a second data center at its Quincy campus, and applied for permits for an addition 13 diesel generators. That’s in addition to the 24 generators currently on site to support Microsoft’s first data center, which uses 48 megawatts of power. The expansion would mean a total of 37 diesel generators, all of them 2.5 megawatt units.

The Washington State Department of Ecology has approved Microsoft’s additional permit, but has also scheduled a public hearing in Quincy on Sept. 28 to hear from residents on the topic. The Ecology department conducted an evaluation of the health risks from diesel engine exhaust particulates, and found that the Microsoft expansion, viewed in isolation, is not likely to impact public health.

State officials and Microsoft are required to appear at a public meeting to present and discuss the generator expansion. The Department of Ecology took the opportunity to seek feedback from area residents, citing the growing concentration of data centers.

“Due to the interest expressed by other data companies to expand or build in the Quincy area, Ecology was concerned that the cumulative effect of diesel engine emissions should be assessed,” the state said in announcing the meeting.

46 Generators in Quincy
Yahoo runs 13 generators at its data center in Quincy, while Intuit has an additional nine gensets, according to an analysis prepared by Gary Palcisko of the Department of Ecology.

“Under most conditions, data center air emissions are minimal,” the report notes, but adds that a “system-wide power failure could result in the simultaneous operation of dozens of large diesel engines.”

While the impact of generators has been minimal thus far, the economic impact of the Microsoft, Yahoo and Intuit projects has been considerable.

Local officials estimate that Grant and Douglas counties have collected $56.4 million in additional sales tax revenue from data center projects between 2006 and 2008. Grant County’s average wage increased 19 percent between 2006 and 2008, while property tax values in the city of Quincy increased by $504 million during that period, according to the Port of Quincy.

“Derived from vegetable oils or animal fat instead of petroleum, biofuel blends can increase water and biological contaminants in fuel supplies,” writes Matt Stansberry. “If handled improperly, biodiesel fuels can stop a data center’s backup systems cold, according to a recent report from the Uptime Institute.” The story relates the experience of UnitedHealth with a usage of biodiesel mix in a Minnesota data center.

Read the full story for more info and some useful resource links.

And sometimes, they serve as a sign of bigger things to come. Last Friday the backup generators arrived at the new Yahoo data center in Lockport, New York. Three 2-megawatt gensets were installed, generating coverage in The Buffalo News.

“The generators, measuring 11 by 46 feet, were lifted off flatbed trucks by a crane supplied by Clark Rigging and Hauling of Lockport,” The News reported. “They were placed atop 7,000-gallon diesel fuel tanks. (Cummins sales engineer Jay) Deshpande said one generator, one tank and a full load of fuel weigh a total of 170,000 pounds. Deshpande said an emergency generator will burn 143 gallons of diesel fuel in an hour. The power is produced by 60-liter, 2,500-horsepower engines.”

Yahoo is building a $150 million data center on a 30-acre property in Lockport, which is expected to  create 125 local jobs paying $65,000 to $75,000 a year once its two-phase construction plan is completed. The first phase of construction is scheduled to be completed in late spring or early summer.

What does a generator installation look like? Here are a couple of videos that can give you an idea:

• Data Cave Installs 2 Megawatt Generator
• Installing a 2.25 Megawatt Diesel Generator

Cummins said the average prices of automatic transfer switches and paralleling systems will remain flat in 2010. Like the company’s generator sets, Cummins Power Generation’s transfer switches and paralleling systems are commonly used in the power infrastructure for data centers and other mission-critical facilities.

See our Generators Channel for more information about diesel generators. For additional video, check out our DCK video archive and the Data Center Videos channel on YouTube.

The price of diesel generators from Cummins Power Generation (CMI) is going up. Cummins said today that it will raise prices by 2 to 7 percent on commercial generator set products between 15 kW and 2700 kW as of Jan. 1, 2009. The company cited “rising commodity and fuel prices as well as the consequence of the current industry dynamics.”

The data center building boom has created backlogs for the large generators that provide emergency backup power, with some facility operators reporting lengthy delays on new units of the most popular 2 megawatt models. With its core market of truck sales lagging, Cummins has benefited from record sales of generators, supported by strong demand from the data center sector.

Digital Realty, which is the world’s largest landlord of data center facilities, said the new service was “unique in the data center industry” and will ensure that its customers can keep their facilities online even during lengthy utility outages.

The scenario is not without precedent. After the Sept. 11 terrorist attacks in New York, the Telehouse carrier hotel facility at 25 Broadway experienced diesel fuel shortages and generator problems that left its customers offline for more than two days. In the 2005 aftermath of Hurricane Katrina, New Orleans colocation provider Zipa had to scramble to find diesel fuel at a time when the city was without power.

Digital Realty Trust (DLR) is seeking to address those type of worst-case scenarios with the agreement with Foster Fuels, which allows Digital’s Turn-Key Datacenter customers to obtain emergency fuel emergency fuel within 24 hours.

The turn-key program, in which Digital Realty delivers finished space to customers, represents a minority of the company’s leased footprint. But the two companies are discussing the possibility of expanding the program to all Digital Realty Trust U.S. facilities.

The agreement allows Digital Realty to arrange for pre-staging of generator fuel to support uninterrupted fuel delivery “even in disaster situations with complex logistics.”

“Digital Realty Trust’s facilities house critical operations for our customers and this agreement with Foster Fuels ensures that customers can stay up and running on generator power even during prolonged utility power outages,” said Ted Martin, Vice President of Technical Operations at Digital Realty Trust. “Foster Fuels has built a very strong reputation for its emergency fuel oil services and it has the resources, operations and protocols to support the specific needs of datacenter customers. We are proud to be the first company in the data center industry to offer this kind of service.”

Foster Fuels has provided emergency fuel support for hospitals, local governments ansd federal agencies such as FEMA during major disasters, including hurricanes Katrina and Rita in 2005. The company said its nationwide operations can provide  fuel support in each of the markets where Digital Realty Trust operates data centers.

“Emergency fuel service is an area we specialize in and we have the contracts, relationships and dedicated equipment to get fuel to our customers when they need it,” said Watt R. Foster Jr., President of Foster Fuels. “In addition to serving Digital Realty Trust and its customers, we are the prime contractor responsible for providing ground fuel support during National Declared Emergencies in support of Department of Homeland Security/FEMA. We have long-term relationships with major oil companies around the country that provide availability of fuel supply from a variety of sources and locations. We also have dedicated equipment, trucks and staff solely for our emergency fuel service. If anybody can make it happen during an emergency, Foster Fuels can.”

Check out our Generator Channel for more articles about diesel generators. For additional video, see our DCK video archive and the Data Center Videos channel on YouTube.