Intel Embraces Submerging Servers in Oil
September 4th, 2012 By: Rich Miller
Will the appetite for ever-more powerful computing clusters push users to new cooling technologies, like submerging servers in liquid coolant? If so, Intel will be ready. The chipmaker is optimizing its technology for servers immersed in oil, an approach that may soon see broader adoption in the high performance computing (HPC) sector.
“We continue to explore server designs, and we’re evaluating how (immersion cooling) can change the way data centers are designed and operated,” said Mike Patterson, senior power and thermal architect at Intel. ”It’s obviously quite a change in mindset.”
Intel has just concluded a year-long test with immersion cooling equipment from Green Revolution Cooling, and affirmed that the technology is highly efficient and safe for servers. The testing, conducted at an Intel data center in New Mexico, may mark a turning point in market readiness for submerged servers, if recent experience with Intel’s embrace of emerging data center designs is any indication.
Eliminates Need for Chillers, Raised Floor
Austin-based Green Revolution says its liquid-filled enclosures can cool high-density server installations for a fraction of the cost of air cooling in traditional data centers. The company says its approach can produce large savings on infrastructure, allowing users to operate servers without a raised floor, computer room air conditioning (CRAC) units or chillers. Green Revolution’s CarnotJet cooling racks are filled with 250 gallons of dielectric fluid, with servers inserted vertically into slots in the enclosure. Fluid temperature is maintained by a pump with a heat exchanger using a standard water loop.
Liquid cooling is used primarily in high-performance computing (HPC) and other applications requiring high density deployments that are difficult to manage with air cooling. Interest in liquid cooling has been on the rise as more applications and services require high-density configurations, prompting data centers to consider infrastructure previously limited to HPC and supercomputing facilities.
The Intel testbed featured two racks of identical servers – one using traditional air cooling and the other immersed in a Green Revolution enclosure. Over the course of a year, the submerged servers had a partial Power Usage Effectiveness (PUE) of 1.02 to 1.03, equaling some of the lowest efficiency ratings reported using that metric. While the PUE measured a limited number of factors compared to other data centers, Intel believes the servers could even more efficient.
Potential Gains from Optimizing for Liquid
“The heat sink in these servers is optimized for air,” said Patterson. “It’s not going to perform as well as if the heat sink is optimized for oil. Just how good could it be? We’ve proven to ourselves that they are efficient. It could mean that a higher clock speed is possible in an oil immersion installation.”
Intel’s interest in alternative cooling designs is driven by growth projections for the high-performance computing sector, which is shaping up as an increasingly important segment of its business. Intel sees 20 percent annual growth for the technical computing market from 2011-2016, and is positioning its Phi line of Many Integrated Core (MIC) offerings as the engine to push new standards in performance.
More power usually means more cooling. That’s why Intel is preparing for immersion cooling. Patterson said that Intel will now develop reference designs and custom motherboards that are optimized for immersion. Intel said it can work directly with HPC customers or with server OEMs to provide equipment optimized for these alternative cooling designs.
JeffPosted September 4th, 2012
Sure, the PUE (or any other efficiency metric) will score these servers extremely well, but that’s not the total cost so it’s almost necessary to build a comparable metric for alternative cooling (since cooling costs are often the largest component of PUE). Looking at the initial investment in liquid-ready components, amortized over the use of the hardware (since the refresh rate is probably 2-4 years like any other compute hardware) plus the additional maintenance costs associated with liquid (plugging leaks, adding time to hardware work, etc) would create a clearer picture of what the real cost is compared to traditional cooling.
The other question is, what happens if you spring a leak…
[...] Intel just concluded a year-long test of the tech with immersion-equipment company Green Revolution Cooling. It has declared that its chips are ready for the dunked-in-oil technique, reports Rich Miller at Data Center Knowledge. [...]
NicholasPosted September 4th, 2012
While there are things that aren’t being considered with the costs, you’re also avoiding the fact that you can potentially have a much greater density within a data center. You can also likely get greater performance for longer while overclocking due to increased dissipation. The rigs I’ve seen successfully built are also known for their longevity. It would be interesting to see if they’re the exception or not.
ScottPosted September 5th, 2012
No mention of how Cray did something very similiar nearly 30 years ago?
[...] Data Center Knowledge Leave A Comment Email This chillers, cooling, green revolution cooling, [...]
[...] you’d expect, they’re using mineral oil, which doesn’t conduct electricity. After a year of testing with Green Revolution Cooling, [...]
UX-adminPosted September 5th, 2012
A processor dies in one of these immersion cooled systems. How would one replace it? How would one replace memory? Disks? Any other field replaceable units?
FrankPosted September 6th, 2012
How long after one removes the motherboard from the oil can one start working on it, to replacing RAM or CPU? How long before it stops dripping?
TonyPosted September 6th, 2012
this gives a new meaning to the word Grease Monkey. as far as how long would one have to wait and replace a component on the mother board, how long does it take to open a vacuum bag PDQ. concerning this repackaged technology, I think it’s a great idea, why waste energy cooling and entire room when the heat sorce can be cooled, in a typical server rack only 50 to 70% of the rack is producing heat, mean while 100% of cooling is being produced and then wasted. I could go on, but know reason to encourage BTU production.
mike smithPosted September 6th, 2012
What happens with the hard drive breather hole? Will it ‘ingest’ oil instead of air? Drop lands on disk, head hits drop, fun ensues.
[...] ‘coolest’ new approach comes from Intel, which has recently completed a year-long trial with Green Revolution Cooling that involves [...]
Wilfred BrimblecombePosted September 9th, 2012
Does oil burn?
ChrisBPosted September 11th, 2012
“What happens with the hard drive breather hole?”
Spinning rust goes in the storage array.
[...] says that Intel has been submerging some of their servers in oil for over a year now. Of course, these aren’t just regular pits of flammable oil; these are extremely specialized [...]
[...] says that Intel has been submerging some of their servers in oil for over a year now. Of course, these aren’t usually unchanging pits of incendiary oil; these are intensely [...]
Mark BPosted September 13th, 2012
Copycats , I did that in 2004 and have the picture to proof it too. My reason, noise of all the fans from overclocking. Run flawless for 2 years 24/7. Then the laptops became fast enough like a regular PC and no noise. That’s when I disassembled my fish-tank, yes, fish-tank.
Actually, I think I had it earlier but that’s when I took the pictures before disassembling. Somewhere is a forum where I showed it all at the time and it was a hot-topic.
I used mineral oil from the drugstore. Of course I bought a 55 Gallon drum of the stuff and run it into my house to the aquarium and then back outside again into the drum via Fish-tank pump. Hard-Drive I made a copper enclosure, sticking out just on top of the aquarium and box was cooled by the oil too. Anything else was submerged, including power supply.
Dr. FfreezePosted September 13th, 2012
Sorry, I couldn’t resist. I did that back in 1999 with 8 gal mineral oil and a 5,000 BTU window AC unit. I got my PC down to -40C.
Good job and cool to see Intel working on this.
[...] Intel Submerging Servers In Oil – Data Center Knowledge Intel has just concluded a year-long test with immersion cooling equipment from Green Revolution Cooling, and affirmed that the technology is highly efficient and safe for servers. [...]
Puget Systems has had DIY instructions for home setup in mineral oil baths since 2007. Enjoy….youtubes floating around as well. PS…a little mineral oil spreads a long way (ie., it’s messy).
Google: mineral-oil puget-systems
Mark B.Posted September 14th, 2012
Dr. FFreeze, I could not remember your online-name, but I remember very well that you where working in a drugstore and got the oil from there and I bought a 55 gallon drum to do the same thing.
You had then a website posting pics, I was setting up the drum.
I taught something is not right with the date of my pictures and it was much earlier then 2004 but could not find an earlier date.
At the time we either e-mailed or where posting somewhere in a forum about the project. Credit to you, because I know you finished your setup about a month earlier then I did.
Great that you posted here because I was racking my brain a few times (whenever I read a article about submerged computers) in all this years what ever happened to you.
Take care Mark
Fascist: I never felt Mineral-Oil was messy, after all it’s used in a lot of skin creams.
[...] ICE.. ) The fact that you are now really starting to get some heavy hitter companies like Intel, 3M and now Facebook pushing the different solutions, starts to make you wonder if the fans and [...]