Microsoft Reveals its Specialty Servers, Racks
REDMOND, Wash. – As it seeks to slash power usage across its global cloud computing platform, Microsoft has been refining its designs for energy-efficient data center hardware. The company is now sharing the details of those designs, which feature customized servers, a high-efficiency power distribution system and in-rack UPS units.
“When Microsoft saw its server counts and data center footprint growing, it became clear that we needed to improve efficiency,” said Dileep Bhandarkar, a Distinguished Engineer at Microsoft, who oversees the company’s server hardware architecture. Bhandarkar began working with server vendors (OEMs) to develop specialized designs based on specs developed by the Microsoft server team. He recently discussed these designs with Data Center Knowledge.
Over the past three years, Microsoft has openly shared information at industry conferences about its data center modules, known as IT-PACs. But it has offered fewer details about the racks and servers inside those containers, limiting disclosures to events for small groups of industry researchers.
Half-Width Server Design
Microsoft’s latest designs for its cloud-scale services are based on a half-width design that fits 2 servers into a 1U slot in a rack. Each server board houses two CPU sockets, with room for 4 DIMM memory slots in each socket. The hardware currently running in Microsoft data centers follow the broad design concepts, but includes customizations by individual vendors and some app-specific modifications. The servers for Bing, Hotmail and Windows Azure reflect different implementations of the design goals.
This design concept allows Microsoft to fit as many as 96 servers into an extra-tall 57U rack, pushing power densities as high as 16 kilowatts per rack. Each rack includes at least two sets of battery packs providing short-term backup power, allowing Microsoft to operate without a central UPS system.
Microsoft also saves energy in its power distribution system, which brings 480V, three-phase AC power directly to the rack, where power supplies convert it to 12V DC power output for the servers. This eliminates power losses from the multiple AC-to-DC conversions common in centralized UPS systems.
In the IT-PAC, the servers have no fans, using air pressure within the module to manage airflow through the servers. That’s an extension of the data center team’s focus on eliminating unnecessary components on each server . “We’ve been doing that for the last three years,” he said. “When I came to Microsoft, every server had a DVD drive.”
Optimizing Server Energy Usage
The Microsoft team optimized its servers to use 40 to 60 watts of power (depending upon whether it uses 4 cores or 6 cores). The design emphasizes high-efficiency components that can operate in expanded ranges of temperature (up to 90 degrees F) and humidity.
Microsoft developed two types of specifications for its server vendors:
- Specs optimized for homogenous deployments of “scale-out” services, such as Bing, Hotmail and Windows Azure. Specs for each of these services are designed for large bulk purchases of servers delivered in pre-populated racks, with new RFPs for each design refinement.
- Specs based on generic standards for an internal catalog offering a dozen different configurations for smaller business groups within Microsoft. The company typically works with a single vendor for 12 to 18 months at a time to maintain continuity.
Bhandarkar says Microsoft has worked with vendors to reduce power use when processors are idle. “It used to be an idle server would be 50 percent of the power (used when the server is active),” he said. “We’ve pushed that down to about 30 percent.”
Another goal is to reduce the power draw from DIMM memory modules. “The DRAM industry is getting the green bug,” said Bhandarkar. “The market is shifting from 1.5V to 1.35V, and heading to 1.2V.”
‘Lots of Opportunities’ for Innovation at the Rack
“We’re working with a lot of the processor vendors, and helping them understand when our server loads look like,” said Bhandarkar. ” There lots of opportunities to optimize rack level power and cooling.”
Shifting the UPS and battery backup functions from the data center into the server cabinet reduces power losses from multiple conversions that occur between the utility power grid and the data center equipment in a standard UPS architecture. “”We did enough analysis to convince people this (rack-level UPS) could be done,” said Bhandarkar.
On the power distribution front, Microsoft examined a number of options, but focused on the power supply. “The entire data center industry revolves around 480 volts and the server industry likes 208 or 415,” said Bhandarkar. “This doesn’t make any sense. The infrastructure for 480V is too big change, so were driving the server industry to change power supplies.”
Part of Broader Industry Conversation
The use of rack-level UPS and streamlined power distribution are similar in concept (if not execution) to cloud-scale system refinements that have been publicly discussed by Google and the Facebook Open Compute project. Bhandarkar says the disclosures are helpful in advancing innovation and best practices in the data center.
“This is not rocket science,” said Bhandarkar. “Smart people facing the same problems will come up with similar solutions. Driving the entire industry forward helps us in the long run.”
While Microsoft has optimized its design, Bhandarkar avoids the word “custom” in describing its process. “I call us a leading-edge adapter,” he said. “Our stuff is not custom. We don’t own the IP (intellectual property). We encourage our vendors to sell it to others so they can recoup their investments. We would like them to sell it to the industry at large. ”
Here’s a look at Microsoft’s design concept for racks and UPS systems. Click for a larger version of the image.
[...] Microsoft Reveals its Specialty Servers, Racks « Data Center Knowledge As it seeks to slash power usage across its global cloud computing platform, Microsoft has been refining its designs for energy-efficient data center hardware. The company is now sharing the details of those designs, which feature custom servers, a high-efficiency power distribution system and in-rack UPS units. (tags: microsoft datacentre architecture) [...]
[...] by Rich Miller on April 25th, 2011 at:http://www.datacenterknowledge.com/archives/2011/04/25/microsoft-reveals-its-specialty-servers-racks… Leave a [...]
StevePosted April 26th, 2011
Why not devote that extra power and cpu cycles to distributed computing initiatives as a charitable write off? Host your site with Azure, help cure cancer!
Gregg McKnightPosted April 26th, 2011
“Why not devote that extra power and cpu cycles to distributed computing initiatives as a charitable write off? Host your site with Azure, help cure cancer!”
Good idea Steve! Coincidently, Microsoft recently launched a program to do just this!
RolandPosted April 27th, 2011
What isn’t reported is the cost to replace the batteries, the single most expensive “consumable item”
Johnny MnemonicPosted May 1st, 2011
That’s a lot of batteries, I think Roland makes a point. Maintenance of them when they hit their MTBF will not be trivial.
Also, the article says “dozen different configurations”: that’s a lot of skus to support. Each requires parts, training. Microsoft should try to slim that down and reduce complexity. With virtualization, they should be able to get more than one customer on a box: one that needs high CPU, and another that needs high disk space, on the same generic box that has both.
I shudder to manage these with Windows, though. I can’t imagine managing a fleet of 100K+ servers with anything other than the CLI. Is there an equivalent to bash, grep, awk, and sed in Windows?
Adrian SevcencoPosted May 3rd, 2011
Isn’t this Half-Width Design of servers a few years old design of supermicro?
[...] It seems shortly after Facebook released their server/data center designs Microsoft has done the same. [...]
Dr. QPosted May 15th, 2011
Take a look at powershell (included in every windows with the .net upgrades) it makes bash and the other *nix tools look like duct tape.
[...] the overall availability of services. Since then, Microsoft has revealed a similar strategy (http://www.datacenterknowledge.com/archives/2011/04/25/microsoft-reveals-its-specialty-servers-racks) as well as Facebook with its promotion of the Open Compute Project (http://opencompute.org/). [...]