A server motherboard being immersed in liquid coolant filling an enclosure from Green Revolution Cooling. Intel just completed a year-long test affirming the effectiveness of the "submerged servers." (Image: Green Revolution)
Will the appetite for ever-more powerful computing clusters push users to new cooling technologies, like submerging servers in liquid coolant? If so, Intel will be ready. The chipmaker is optimizing its technology for servers immersed in oil, an approach that may soon see broader adoption in the high performance computing (HPC) sector.
"We continue to explore server designs, and we're evaluating how (immersion cooling) can change the way data centers are designed and operated," said Mike Patterson, senior power and thermal architect at Intel. "It's obviously quite a change in mindset."
Intel has just concluded a year-long test with immersion cooling equipment from Green Revolution Cooling, and affirmed that the technology is highly efficient and safe for servers. The testing, conducted at an Intel data center in New Mexico, may mark a turning point in market readiness for submerged servers, if recent experience with Intel's embrace of emerging data center designs is any indication.
Eliminates Need for Chillers, Raised Floor
Austin-based Green Revolution says its liquid-filled enclosures can cool high-density server installations for a fraction of the cost of air cooling in traditional data centers. The company says its approach can produce large savings on infrastructure, allowing users to operate servers without a raised floor, computer room air conditioning (CRAC) units or chillers. Green Revolution’s CarnotJet cooling racks are filled with 250 gallons of dielectric fluid, with servers inserted vertically into slots in the enclosure. Fluid temperature is maintained by a pump with a heat exchanger using a standard water loop.
Liquid cooling is used primarily in high-performance computing (HPC) and other applications requiring high density deployments that are difficult to manage with air cooling. Interest in liquid cooling has been on the rise as more applications and services require high-density configurations, prompting data centers to consider infrastructure previously limited to HPC and supercomputing facilities.
The Intel testbed featured two racks of identical servers - one using traditional air cooling and the other immersed in a Green Revolution enclosure. Over the course of a year, the submerged servers had a partial Power Usage Effectiveness (PUE) of 1.02 to 1.03, equaling some of the lowest efficiency ratings reported using that metric. While the PUE measured a limited number of factors compared to other data centers, Intel believes the servers could even more efficient.
Potential Gains from Optimizing for Liquid
"The heat sink in these servers is optimized for air," said Patterson. "It's not going to perform as well as if the heat sink is optimized for oil. Just how good could it be? We've proven to ourselves that they are efficient. It could mean that a higher clock speed is possible in an oil immersion installation."
Intel's interest in alternative cooling designs is driven by growth projections for the high-performance computing sector, which is shaping up as an increasingly important segment of its business. Intel sees 20 percent annual growth for the technical computing market from 2011-2016, and is positioning its Phi line of Many Integrated Core (MIC) offerings as the engine to push new standards in performance.
More power usually means more cooling. That's why Intel is preparing for immersion cooling. Patterson said that Intel will now develop reference designs and custom motherboards that are optimized for immersion. Intel said it can work directly with HPC customers or with server OEMs to provide equipment optimized for these alternative cooling designs.
Merits of Mineral Oil
Mineral oil has been used in immersion cooling because it is not hazardous and transfers heat almost as well as water, but doesn’t conduct an electric charge. Some mineral oil-style coolants can be messy to maintain. Patterson said Intel's technicians always had an extra set of clothes handy to account for coolant spills. Green Revolution says its coolant can be drained for enclosure-level maintenance, and individual servers can be removed for work.
"I think it will catch on,” said Patterson. "It's going to be a slow progression, but will start in high-performance computing and the need to look at performance upside. Those (HPC) procurements are large enough that this might make sense. Those guys are probably a little bit more sophisticated in their willingness to do something different.”
Green Revolution says its GreenDEF coolant can support heat loads of up to 100 kilowatts per 42U rack, far beyond current average heat loads of 4 to 8 watts a rack and high-density loads of 12 to 30 kilowatts per rack. The company says the system is designed to comply with fire codes and the Clean Water Act, and integrates with standard power distribution units (PDUs) and network switches.
Gains for Green Revolution
Patterson emphasized that Intel doesn't recommend a particular vendor. "We are evaluating all types of oil immersion to understand the entire ecosystem," said Patterson. Other players in the immersion cooling field include Iceotope and LiquidCool (formerly Hardcore Computer).
But there's no question that the cooperation with Intel is an important validation for Green Revolution's technology, which we've been tracking since the company's launch in 2009. The technology is being used by an Austin colocation provider, as well as in HPC installations hwere it has demonstrated its ability to work with GPU acceleration and designs that recycle server waste heat.
This week Green Revolution announced that server maker Super Micro Computer is providing full warranty coverage for servers installed in the CarnotJet submersion system, which the company called a "momentous step forward" for its technology. The warranty will cover motherboards, backplanes, add-on cards, power supplies, and processors.
More broadly, submersion cooling may benefit from Intel's affirmation that submersion doesn't have any impact on hardware reliability. "When we pulled the servers out, we took them to our failure analysis lab," Patterson said. "They could find nothing at all to suggest that this is a bad idea."
In 2008, Intel published research showing that in many climates servers perform well when cooled with fresh air from outside the data center. At the time, this design approach - known as air-side economization or "free cooling" was used by a small percentage of data centers. With support from Intel and other large data center builders, fresh air cooling has since become a common feature in new data center construction in recent years.
A four-rack installation of the Green Revolution liquid coolings solution, which submerges servers in a coolant similar to mineral oil. (Photo: Green Revolution)
