A graphic of the new data center power distribution system being implemented at Facebook, which replaces a central UPS with a battery built into the power supply.

Facebook Follows Google to Data Center Savings

18 comments

A graphic of the new data center power distribution system being implemented at Facebook, which replaces a central UPS with a battery built into the power supply.

A graphic of the new data center power distribution system being implemented at Facebook, which replaces a central UPS with a battery built into the power supply.

Facebook is stepping up its efforts to make its data centers cheaper and more efficient, and is following in the footsteps of Google in several of its key initiatives. Facebook says it’s streamlining its servers, and also plans to adopt a novel power distribution design pioneered by Google.

The social network’s plans were discussed by Amir Michael, a server and data center engineer at Facebook, in a Nov. 17 engineering Tech Talk in Palo Alto, Calif. Michael joined Facebook in March after six years on the data center team at Google, where he designed cooling and electrical systems.

“The industry has discussed many ways of optimizing servers and data centers,” Michael said. “We don’t claim to be the original thought leader on many of these things, but we are gonna be one of the few companies who’s actually going to begin implementing them.”

Shift to On-Board Battery
That’s true of the most significant change outlined by Michael: a new power distribution design that removes traditional uninterruptible power supply (UPS) and power distribution units (PDUs) from the data center. The new design shifts the UPS and battery backup functions from the data center into the cabinet, adding a 12 volt battery to each server power supply.

In April, Google revealed its use of a power supply that integrates a 12 volt battery, allowing it to function as an on-board UPS. The company cited this design as a key factor in the exceptional energy efficiency data it has reported for its data centers, including Power Usage Effectiveness (PUE) ratings between 1.1 and 1.2.

Major Efficiency Gains
Facebook says it expects to gain similar efficiency benefits, reducing the energy loss during power distribution from the current 35 percent to about 15 percent. The company said it can lower its power bill by simplifying how electricity travels to its servers. 

In most data centers, a UPS system stands between the utility power grid and the data center equipment. When there is a grid outage, the UPS taps a large bank of batteries (or in some cases, a flywheel) for “ride-through” power until the generator can be started. The AC power from the grid is converted into DC power to charge the batteries, and then converted back to AC for the equipment.

Saving on Equipment
In addition to improving efficiency, the new design could yield enormous savings on equipment purchases. “You no longer need to buy a traditional UPS and PDU system,” said Michael. “On the capex side, it’s a huge win. This can save millions of dollars that you no longer have to spend on a UPS system. We hope to see the industry move to a model like this.”

Facebook’s enormous growth is clearly giving it leverage with its vendors, which are working with the company as it customizes its equipment. An example: typical servers use 208 volt power to the servers. Most power supplies can also support the 230 volt and 240 volt configurations now being implemented to capture extra efficiency.

Facebook’s new distribution scheme calls for 277 volt power to the servers. “We’re working with power supply vendors to create a (server) power supply that will accept 277 volts on the input,” said Michael. 

Stripped-Down Servers
Michael said Facebook is also stripping down servers, noting that most servers are tailored for enterprise customers, and include cosmetic features and extras that can impede air flow for cooling. 

“I would prefer to have an ugly server,” said Michael. “An ugly server is a beautiful server because I paid less money for it. We can remove a lot of connectors and a lot of cost from the motherboard itself. This can come in the form of removing video cards, removing extra disk connectors we don’t need, (and) removing PCIe connectors which really don’t provide us with any functionality.”

Few Details on Timeline
Michael didn’t provide details on Facebook’s progress in implementing these new approaches, but the power distribution plans were discussed in the future tense. Michael also outlined the company’s plans to make greater use of free cooling, which uses fresh air rather than chilled water to cool servers. Facebook is known to be using free cooling in at least one of its data centers.

Facebook also said it intends to raise the temperature in its data centers, a strategy that expands the number of hours in which you can use free cooling. This is a strategy that is being widely implemented by large data center users, including Google. “Because we’re taking more ownership of our server design, we can now design it in a way which allows us to raise the temperature in the data center,” Michael said.

RELATED STORIES:

About the Author

Rich Miller is the founder and editor at large of Data Center Knowledge, and has been reporting on the data center sector since 2000. He has tracked the growing impact of high-density computing on the power and cooling of data centers, and the resulting push for improved energy efficiency in these facilities.

Add Your Comments

  • (will not be published)

18 Comments

  1. Rackable Systems

    Stunning that Facebook can brush us off for years as we showed them specifically how to put batteries into their servers and the rack and server level as well as offering them a machine with a low linear-foot-cooling so that they could raise the temperatures in their datacenters. Even gave them the models for how the heat dissipation would work. And now, finally, they have "invented" the exact same thing.

  2. Having UPS systems sitting in-line down-converting 120VAC to 12VDC, then up-converting 12VDC to 120VAC only to have it down-converted again back to 12VDC was a practice of convenience but certainly defied logic when speaking of efficiency. It's good to see systems where batteries are put on standby trickle charge and connected directly to the equipment as backup bypassing that wole nonsense of down then up then down again. In terms of "Rackable Systems", you just got to groan and bear it as there's a lot of that re-invented going around.

  3. Rackable - if you have a case, sue them. They probably wanted to stay away from vendor lock -real or perceived - for whatever reason. Anyway, I have been wondering when the chasm will be crossed for these single tenant applications/deployments into the multi tenant realm. Containers are starting to (although still most prevalent in single tenant operations), owever when we point to these 'revolutionary' developments, I believe the revolution will be when it's generally available for the rest of us.

  4. Cellar

    In one sense, we've come full circle. Run a system that doesn't require a video card, and you can manage it through a serial port. That has long been standard practice before the wintendo came along. Now we're going back there, only with the serial grafted onto a system that never took to it very well. But such is the way of mainstream computing. Wonder how long it'll take the big'uns to catch up to more efficient CPU architectures, too.

  5. CloudCEO

    I wonder how long until we see a cost effective CPU fabric that doesn't need to be cobbled together with customized parts. Just CPU, I/O, network, Disk, Power and management. Probably could fit it in half trhe space and use 1/3rd the power and cooling.

  6. Sam

    @Cellar as if any large numbers of servers at Google and Facebook run windows.

  7. anon

    Um.. guys.. that article about Google they linked to was published on April 1st... April fools anyone???

  8. Um, guys ... You had to be anonymous for that? The announcements from the Data Center Summit were plenty real. We noted the oddness of the April 1 scheduling, but the summit was a day long event witnessed by many of the world's leading data center experts. For what it's worth, Google also introduced Gmail on an April 1. I'm pretty sure Gmail is real.

  9. Peter Buck

    I'm finding it frustrating that no overall efficiency gain was given. Since data centers are a significant contributor to the world power budget (over 1% in the US and surely well above that worldwide), potential efficiency reductions like this are welcome news, especially since the power requirements for data centers are projected grow by more than an order of magnitude by 2020 (http://www.greentechmedia.com/green-light/post/data-center-power-consumption-by-the-numbers-341/).

  10. Both Google and Facebook have taken the approach of propagating the most prominent point of failure (batteries) onto every server, which is problematic because the batteries fail regularly without notice. In order to make this approach work, a few key features need to be in place: (1) there needs to be significant redundancy in the software systems to ensure that battery or power failures don’t degrade the services provided; (2) there needs to be a strong service program in place to check, test and replace batteries that are no longer functioning; and (3) batteries have high sensitivity to temperatures, so any application needs to consider the tight control of temperatures for the battery application.

  11. data center operator

    my quick take on this is that it defies physics to expect a bunch of very small battery systems (associated with small transformers, rectifiers, etc) to peform more efficiently that one (or two, if 2N) large scale systems. scale drives efficiency - smallish & distributed designs drive inefficiency. also, i totally agree with last post that batteries are an especially troublesome point of failure. this approach would make no sense but for virtualization, where in theory, the loss of a single physical server can be absorbed due to logical failover to other servers. in the real world, many or all DCK readers know that it doesnt always work that way and the best way to avoid cascading failures is to avoid the first one

  12. In regards to Gary A., proper battery monitoring must be implemented into the data centers to protect from battery failure. Eagle Eye's known for 24/7 battery monitoring that reduces most of the service needed in testing and replacing batteries.