September 27th, 2010 By: Rich Miller
We continue with the Facebook Data Center FAQ (or Everything You Ever Wanted to Know About Facebook’s Data Centers).
How Many Servers Does Facebook Have?
“When Facebook first began with a small group of people using it and no photos or videos to display, the entire service could run on a single server,” said Jonathan Heiliger, Facebook’s vice president of technical operations.
Not so anymore. Facebook doesn’t say how many web servers it uses to power its infrastructure. Technical presentations by Facebook staff suggest that as of June 2010 the company was running at least 60,000 servers in its data centers, up from 30,000 in 2009 and 10,000 back in April 2008.
There are companies with more servers (see Who Has the Most Web Servers? for details). But the growth curve shown on the chart doesn’t even include any of the servers that will populate the Oregon data center – which may be the first of multiple data centers Facebook builds to support its growth.
What kind of servers does Facebook use?
In a marked departure from industry practice, Facebook has disclosed the designs and specs for its data centers. In April 2011 the social network launched the Open Compute Project, through which it is releasing the details of its energy efficient data center design, as well as its custom designs for servers, power supplies and UPS units.
Facebook’s servers are powered by chips from both Intel and AMD, with custom-designed motherboards and chassis built by Quanta Computer of Taiwan. The servers use a 1.5U (2.65 inch) chassis, allowing the use of larger heat sinks and fans to improve cooling efficiency.
“We removed anything that didn’t have a function,” said Amir Michael, a hardware Engineer at Facebook. “No bezels or paints. The slightly taller chassis allowed us to use taller heat sinks. We’re also able to use larger 60 millimeter fans rather than 40 millimeter fans. The 60 millimeter fans are more efficient.”
The cabling and power supplies are located on the front of the servers, so Facebook staff can work on the equipment from the cold aisle, rather than the enclosed, 100-degree plus hot aisle.
Facebook’s servers include custom power supplies that allow servers to use 277-volt AC power instead of the traditional 208 volts. This allows power to enter the building at 400/277 volts and come directly to the server, bypassing the step-downs seen in most data centers as the power passes through UPS systems and power distribution units (PDUs). The custom power supplies were designed by Facebook and built by Delta Electronics of Taiwan and California-based Power One.
Facebook contemplated installing on-board batteries on its servers, but settled on in-row UPS units. Each UPS system houses 20 batteries, with five strings of 48 volt DC batteries. Facebook’s power supplies include two connections, one for AC utility power and another for the DC-based UPS system. The company has systems in place to manage surge suppression and deal with harmonics (current irregularities).
What kind of software does Facebook Use?
Facebook was developed from the ground up using open source software. The site is written primarily in the PHP programming language and uses a MySQL database infrastructure. To accelerate the site, the Facebook Engineering team developed a program called HipHop to transform PHP source code into C++ and gain performance benefits.
Facebook has one of the largest MySQL database clusters anywhere, and is the world’s largest users of memcached, an open source caching system. Memcached was an important enough part of Facebook’s infrastructure that CEO Mark Zuckerberg gave a tech talk on its usage in 2009.
Facebook has built a framework that uses RPC (remote procedure calls) to tie together infrastructure services written in any language, running on any platform. Services used in Facebook’s infrastructure include Apache Hadoop, Apache Cassandra, Apache Hive, FlashCache, Scribe, Tornado, Cfengine and Varnish.