Clustered Systems Cools 100kW in Single Rack
December 12th, 2011 By: Rich Miller
A new blade server chassis featuring technology from Clustered Systems is promising to cool computing loads of up to 100 kilowatts in a single cabinet. The system, which breaks new ground in the effort to pack massive computing power into smaller spaces, will get its first test drive at the SLAC National Accelerator Laboratory in Palo Alto, Calif.
Average server racks in most data centers use between 4 kilowatts (kW) and 8 kW of power. Cloud computing and high-performance computing (HPC) centers feature denser infrastructures of 12kW to 20 kW and more. The new blade chassis promises to push the boundaries of high-density computing to 80kW to 100kW per rack.
Perhaps most intriguing: the system requires only a 480V power source, and a water supply, with no chillers and minimal cooling infrastructure.
“Changing the Data Center Dynamic”
“If we are successful, then the whole dynamic of data center deployment could change,” said Phil Hughes, CEO and founder of Clustered Systems. “A user can put a system anywhere there is power. No special facilities are required. All investment can go into compute and not have to be shared with bricks and mortar.”
The new blades build on Clustered Systems’ success in a 2010 “chill-off” in which its technology proved more efficient than existing cooling products from major data center vendors.
They key to the system’s density is a fanless cooling system using a cold plate, which contains a tubing system filled with liquid refrigerant. By removing fans and dedicating more power to processors, the Clustered Systems design can support unusual power densities.
The refrigerant system includes a pumping unit and heat exchanger, in which the refrigerant interacts with a water loop. In testing, the system has continued working with water temperatures as high as 78 degrees, meaning it can operate without a chiller, according to Hughes.
“It is expected that the initial deployment will be cooled with tower water or with return water from upstream legacy cooling systems,” he said.
Consortium of Partners
In 2010 Clustered Systems partnered with Emerson Network Power on the Liebert XDS system, which used cold plates on server trays in a 1U rackmount design. The installation at SLAC adapts the technology for blade servers, which can be challenging to cool due to the way they concentrate processing and power consumption.
Each chassis takes up 8 rack units and includes 16 blades, each with two cold plates for heat removal, and a 20kW power distribution unit. Five of the 8U chassis can fit in a rack.
The blade server chassis was jointly developed by a group of companies including Clustered Systems, Intel, Emerson Network Power, Panduit, OSS (One Stop Systems, Inc.), Smart Modular and Inforce. The system development was funded by $3 million in grants from the U.S. Department of Energy and California Energy Commission.
“The efficiency of the Clustered Systems’ cooling system supports the greatest level of density and performance we’ve seen so far, and it has the legs to support several more product generations,” said Dr. Stephen Wheat, Senior Director of Intel High Performance Computing.
The cooling system uses Emerson Network Power’s Liebert XD pumped refrigerant cooling products. Emerson also designed and built the system rack, which features a NetSure DC power system which converts 480V AC power to 380V DC power.
The 380V DC will then pass to a Panduit unit in each enclosure which controls power delivery to each blade. “The concept of a power plane manufactured into the cabinet can be a source of improved efficiency in the data center,” said Jack Tison CTO, Panduit, Inc. The 380V DC is then converted to 12V DC at the chassis level.
The dual CPU modules use PCI express as the system interconnect for its network, which was developed by One Stop Systems. “All blades in a system communicate with each other at 40Gb/s over PCI Express (PCIe), increasing the overall performance of the system,”said Stephen Cooper, CEO of OSS. “By utilizing the inherent functionality of PCIe over cable, we’ve designed switch blades and large 40-port switches that provide complete non-blocking communication at previously unheard of performance rates.”
In the chassis, each blade houses two motherboards, each with two processors from the future Intel Xeon E5 family. The motherboards were designed by Inforce. The DIMM memory modules were designed as a cooperative effort between SMART Modular and Clustered. “These modules are a derivative of standard DIMMs and include an optimized heatsink design that creates an efficient and cost effective method to transfer heat from the DIMMs to the cold plate,” said Mike Rubino SMART Modular’s VP of Engineering.
The first two racks are scheduled to be installed at Stanford Linear Accelerator (SLAC) within the next few months. For the cooling system, SLAC will use cooling water exiting from existing IT equipment or directly from a cooling tower.
“We are very excited to be chosen as the first deployment site,” said Norm Ringold, Head of IT Operations and Infrastructure, SLAC National Accelerator Laboratory. “The estimated 50 Teraflops per rack will add considerably to our compute capacity,”
Clustered Systems has not announced detailed pricing, but says it will be “highly competitive with other POD and container based systems.” The company says a 3.2 megawatt data center using the new blade chassis could cost as little as $9.2 million, or about $3 million per megawatt of critical load. Industry experts new data center construction costs about $10 million to $15 million per megawatt on average, with hyper-scale projects like those at Google and Yahoo slashing that to $5 million per megawatt.
Hughes says the target market will begin on a slightly smaller scale.
“The ideal customer will have need for HPC but no data center space to house it,” he said. “Typical customers could be academic department heads with money for hardware but not for infrastructure, or high frequency traders wanting to maximize crunch power in a very small allocated space. Longer term, we also expect to address cloud computing, which has much the same requirements as HPC.”
Seems like hype… 4 sockets per blade / 32 servers in 8U, that’s 160 servers in a rack. There’s just not enough space to put stuff in that will draw anywhere remotely to 100kW in a 42-47U rack.
Supermicro with their twin blade design can get 120 servers in a rack with their twinblade, HP – 128 servers with their double density blades.
I checked the power consumption of a maxed out HP blade system less than 31kW for 128 blades with every expansion slot on every system in use, comes to an average of 242W/server.
Since everyone is using pretty much the same things that drive the power on these boxes (CPU, RAM) with 242W/server, you’d need a way to pack 400 servers into a rack to draw 100kW of power. Good luck doing that, even in a shipping container with 57U racks.
The 100kW number makes it sound like this company is years ahead, while technically they may be able to cool that number the product they have now isn’t as competitive from a density perspective as the 100kW number leads one to believe.
This is of course assuming you can actually draw 100kW and is not a claim of 50kW active / 50kW passive since that would not be “cooling 100kW in a single rack”
The 100KW is the power and cooling design point. Even though the design point for today’s half width server board is only 450W thus the rack load 72KW. However, CPUs are expected to get faster and draw even more power while GPUs can already consume up to 300W per unit. Having the extra headroom now offers a degree of future proofing.