Dunking for Density: New Projects Pursue 3M’s Take on Immersion Cooling
November 13th, 2013 By: Rich Miller
We’re continuing to see new examples of immersion cooling at meaningful scale. In July we brought you an update on an immersion cooling system at CGG using technology from Green Revolution Cooling. We’ve also been tracking early projects using “open bath immersion” cooling based on technology developed by 3M.
Open bath immersion (OBI) is an example of passive two-phase cooling, which uses a boiling liquid to remove heat from a surface and then condenses the liquid for reuse, all without a pump. The servers are immersed in 3M’s Novec, a non-conductive chemical with a very low boiling point, which easily condenses from gas back to liquid. The OBI technique, which we first saw at last year’s Data Center World show, is now in use in a handful of sites. Here’s an overview of some of these projects.
Hong Kong-based Allied Control is specializing in developing high-density cooling solution using 3M’s Novec and OBI. The company has recently deployed a 500kW high performance computing (HPC) production installation known as Immersion-2 for a client in Hong Kong. This design uses OBI in standard 19-inch racks, and was deployed in less than six months. Allied Control says the system operates at a Power Usage Effectiveness (PUE) of 1.02, which would make it one of the most efficient designs in the world, even though Hong Kong has a hot and humid climate. The facility is located in a high rise building and fits in the size of a standard shipping container.
One issue with a rack-mounted approach to immersion cooling is weight. “The weight is indeed a small challenge for standard sized racks, but actually more due to the increased system density, not really the fluid,” said Alex Kampl, VP of Engineering for Allied Control. “You also remove a lot of weight by not using air cooling. We’ve been working with a rack manufacturer who has been very helpful.” Here’s a look at the facility:
The company previously built a dedicated immersion-cooled facility called Immersion-1 to cool a unique supercomputer comprised of FPGAs (field programmable gate arrays), which are semiconductor devices that can be programmed after manufacturing.
Allied Control created Immersion-1, a system using 6,048 FPGA chips combining 890 million logic cells, which will encompass up to 24 tanks. The company says that a similar installation using traditional air cooling would require more than 8,500 Dual Xeon 1U servers in more than 200 racks.
“Immersion-1 has become a massive prototype and proof of concept for a whole new generation of computing,” Allied Control says on its web site. “Since the special application tweaks the maximum performance out of each FPGA, they generate much more heat than in traditional FPGA applications. Often, FPGAs have to be throttled down or are not running at maximum performance due to cooling issues. In case of Immersion-1, the cluster would not be able to run on passive cooling and the FPGA chip temperature rises above its maximum specifications within seconds. Only by using immersion cooling it was possible to build and run Immersion-1 with its very demanding cooling requirements.”
The company has also developed a design concept to adapt Intel’s Xeon Phi coprocessor for HPC workloads in immersion cooling, and is interested in developing high-density designs using Intel’s Dense Form Factor (DFF) cards in open bath immersion.
“OBI is in its early stages, but I am sure we’ll see exciting progress very soon,” said Kampl. “Unfortunately we are wasting a lot of time right now to literally remove unnecessary parts from hardware built for air-cooling, so my hope is that system designers start offering similar hardware like the DFF cards.”
The Allied Control technology will be on display at the 3M booth at next week’s SC13 conference in Denver.
There are no plans to have an immersion system at LBNL. We are working on the system you report for the Navy Research Lab only.
MikePosted November 28th, 2013
Them thar servers you see are mining all teh bitcoins!
all your bitcoin are belong to them.
[…] again, removing heat from the components in the process. The ASICMINER open bath immersion system was reportedly built by Hong Kong-based company Allied Control, and operates at a Power Usage Effectiveness of 1.02, which “would make it one of the most […]