Dunking for Density: New Projects Pursue 3M’s Take on Immersion Cooling

3 comments

3m-applied-cooling

This immersion cooling project in Hong Kong was created by Allied Control using a two-phase cooling technique called open bath immersion (OBI), using 3M’s Novec fluid.

We’re continuing to see new examples of immersion cooling at meaningful scale. In July we brought you an update on an immersion cooling system at CGG using technology from Green Revolution Cooling. We’ve also been tracking early projects using “open bath immersion” cooling based on technology developed by 3M.

Open bath immersion (OBI) is an example of passive two-phase cooling, which uses a boiling liquid to remove heat from a surface and then condenses the liquid for reuse, all without a pump. The servers are immersed in 3M’s Novec, a non-conductive chemical with a very low boiling point, which easily condenses from gas back to liquid. The OBI technique, which we first saw at last year’s Data Center World show, is now in use in a handful of sites. Here’s an overview of some of these projects.

Allied Control

Hong Kong-based Allied Control is specializing in developing high-density cooling solution using 3M’s Novec and OBI. The company has recently deployed a 500kW high performance computing (HPC) production installation known as Immersion-2 for a client in Hong Kong. This design uses OBI in standard 19-inch racks, and was deployed in less than six months. Allied Control says the system operates at a Power Usage Effectiveness (PUE) of 1.02, which would make it one of the most efficient designs in the world, even though Hong Kong has a hot and humid climate. The facility is located in a high rise building and fits in the size of a standard shipping container.

One issue with a rack-mounted approach to immersion cooling is weight. “The weight is indeed a small challenge for standard sized racks, but actually more due to the increased system density, not really the fluid,” said Alex Kampl, VP of Engineering for Allied Control. “You also remove a lot of weight by not using air cooling. We’ve been working with a rack manufacturer who has been very helpful.” Here’s a look at the facility:

These racks are filled with high-density servers immersed in cooling fluid. The installation in Hong Kong was created by Allied Control using 3M's Novec fluid. (Photo: Allied Control)

These racks are filled with tanks containing high-density servers immersed in cooling fluid. The installation in Hong Kong was created by Allied Control using 3M’s Novec fluid. (Photo: Allied Control)

Inside the rack-mounted tanks, the heat from dozens of servers causes the Novec fluid to boil. The vapor cools when it reaches the condenser at the top of the tank and is then reused. (Photo: Allied Control)

Inside the rack-mounted tanks, the heat from dozens of servers causes the Novec fluid to boil. The vapor cools when it reaches the condenser at the top of the tank and is then reused. (Photo: Allied Control)

The company previously built a dedicated immersion-cooled facility called Immersion-1 to cool a unique supercomputer comprised of FPGAs (field programmable gate arrays), which are semiconductor devices that can be programmed after manufacturing.

Allied Control created Immersion-1, a system using 6,048 FPGA chips combining 890 million logic cells, which will encompass up to 24 tanks. The company says that a similar installation using traditional air cooling would require more than 8,500 Dual Xeon 1U servers in more than 200 racks.

“Immersion-1 has become a massive prototype and proof of concept for a whole new generation of computing,” Allied Control says on its web site. “Since the special application tweaks the maximum performance out of each FPGA, they generate much more heat than in traditional FPGA applications. Often, FPGAs have to be throttled down or are not running at maximum performance due to cooling issues. In case of Immersion-1, the cluster would not be able to run on passive cooling and the FPGA chip temperature rises  above its maximum specifications within seconds. Only by using immersion cooling it was possible to build and run Immersion-1 with its very demanding cooling requirements.”

The company has also developed a design concept to adapt Intel’s Xeon Phi coprocessor for HPC workloads in immersion cooling, and is  interested in developing high-density designs using Intel’s Dense Form Factor (DFF) cards in open bath immersion.

“OBI is in its early stages, but I am sure we’ll see exciting progress very soon,” said Kampl. “Unfortunately we are wasting a lot of time right now to literally remove unnecessary parts from hardware built for air-cooling, so my hope is that system designers start offering similar hardware like the DFF cards.”

The Allied Control technology will be on display at the 3M booth at next week’s SC13 conference in Denver.

Next: Projects by 3M, Lawrence Berkeley Lab

Pages: 1 2

About the Author

Rich Miller is the founder and editor at large of Data Center Knowledge, and has been reporting on the data center sector since 2000. He has tracked the growing impact of high-density computing on the power and cooling of data centers, and the resulting push for improved energy efficiency in these facilities.

Add Your Comments

  • (will not be published)

3 Comments

  1. There are no plans to have an immersion system at LBNL. We are working on the system you report for the Navy Research Lab only.

  2. Mike

    Them thar servers you see are mining all teh bitcoins! all your bitcoin are belong to them.