The Facebook Data Center FAQ (Page 2)

We continue with the Facebook Data Center FAQ (or Everything You Ever Wanted to Know About Facebook’s Data Centers).

How Many Servers Does Facebook Have?

This chart provides a dramatic visualization of Facebook’s infrastructure growth. It documents the number of servers used to power Facebook’s operations. 

“When Facebook first began with a small group of people using it and no photos or videos to display, the entire service could run on a single server,” said Jonathan Heiliger, Facebook’s vice president of technical operations.

Not so anymore. Technical presentations by Facebook staff suggest that as of June 2010 the company was running at least 60,000 servers in its data centers, up from 30,000 in 2009 and 10,000 back in April 2008.

There are companies with more servers (see Who Has the Most Web Servers? for details). But the growth curve shown on the chart doesn’t even include any of the servers that will populate the Oregon data center – which may be the first of multiple data centers Facebook builds to support its growth.    

What kind of servers does Facebook use?
Facebook doesn’t often discuss which server vendors it uses. In 2007 it was buying a lot of servers from Rackable (now SGI), and is also known to have purchased servers from Dell, which customizes servers for its largest cloud computing customers.

Facebook VP of Technical Operations Jonathan Heiliger has sometimes been critical of major server vendors’ ability to adapt their products to the needs of huge infrastructures like those at Facebook, which don’t need many of the features designed for complex enterprise computing requirements. “Internet scale” companies can achieve better economics with bare bones servers that are customized for specific workloads. 

In a conference earlier this year, Heiliger identified multi-core server vendors Tilera and SeaMicro as “companies to watch” for their potential to provide increased computing horsepower in a compact energy footprint.

But reports that Facebook planned to begin using low-power processors from ARM - which power the iPhone and many other mobile devices - proved to be untrue. “Facebook continuously evaluates and helps develop new technologies we believe will improve the performance, efficiency or reliability of our infrastructure,” Heiliger said. “However, we have no plans to deploy ARM servers in our Prineville, Oregon data center.”

A look at the fully-packed server racks inside a Facebook data center facility.

What kind of software does Facebook Use?
Facebook was developed from the ground up using open source software. The site is written primarily in the PHP programming language and uses a MySQL database infrastructure. To accelerate the site, the Facebook Engineering team developed a program called HipHop to transform PHP source code into C++ and gain performance benefits.

Facebook has one of the largest MySQL database clusters anywhere, and is the world’s largest users of memcached, an open source caching system. Memcached was an important enough part of Facebook’s infrastructure that CEO Mark Zuckerberg gave a tech talk on its usage in 2009.

Facebook has built a framework that uses RPC (remote procedure calls) to tie together infrastructure services written in any language, running on any platform. Services used in Facebook’s infrastructure include Apache Hadoop, Apache Cassandra, Apache Hive, FlashCache, Scribe, Tornado, Cfengine and Varnish.

How much Does Facebook Spend on Its Data Centers?
An analysis of Facebook’s spending with data center developers indicates that the company is now paying about $50 million a year to lease data center space, compared to about $20 million when we first analyzed its leases in May 2009.

The $50 million a year includes spending is for leases, and doesn’t include the cost of the Prineville project, which has been estimated at between $180 million and $215 million. It also doesn’t include Facebook’s investments in server and storage hardware, which is substantial.

Facebook currently leases most of its data center space from four companies: Digital Realty Trust, DuPont Fabros Technology, Fortune Data Centers and CoreSite Realty.

Here’s what we know about Facebook’s spending on its major data center commitments:

• Facebook is paying $18.1 million a year for 135,000 square feet of space in data center space it leases from Digital Realty Trust (DLR) in Silicon Valley and Virginia, according to data from the landlord’s June 30 quarterly report to investors.
• The social network is also leasing data center space in Ashburn, Virginia from DuPont Fabros Technology(DFT). Although the landlord has not published the details of Facebook’s leases, data on the company’s largest tenants reveals that Facebook represents about 15 percent of DFT’s annualized base rent, which works out to about $21.8 million per year.
• Facebook has reportedly leased 5 megawatts of critical load – about 25,000 square feet of raised-floor space – at a Fortune Data Centers facility in San Jose.
• In March, Facebook agreed to lease an entire 50,000 square foot data center that was recently completed by CoreSite Realty in Santa Clara.
• Facebook also hosts equipment in a Santa Clara, Calif. data center operated by Terremark Worldwide (TMRK), a Palo Alto, Calif. facilityoperated by Equinix (EQIX) and at least one European data center operated by Telecity Group. These are believed to be substantially smaller footprints than the company’s leases with Digital Realty and DuPont Fabros.

That adds up to an estimated $40 million for the leases with the Digital Realty and DuPont Fabros, When you add in the cost of space for housing equipment at Fortune, CoreSite, Terremark, Switch and Data, Telecity and other peering arrangements to distribute content, we arrive at an estimate of at least $50 million in annual data center costs for Facebook.

Facebook’s costs remain substantially less than what some other large cloud builders are paying for their data center infrastructure. Google spent $2.3 billion on its custom data center infrastructure in 2008, while Microsoft invests $500 million in each of its new data centers. Those numbers include the facilities and servers.

How Many People Are Needed to Run Facebook’s Data Centers?
As is the case with most large-scale data centers, Facebook’s facilities are highly automated and can be operated with a modest staff, usually no more than 20 to 50  employees on site. Facebook has historically maintained a ratio of 1 engineer for every 1 million users, although recent efficiencies have boosted that ratio to 1 engineer for every 1.2 million users.

Facebook’s construction project in Prineville is expected to create more than 200 jobs during its 12-month construction phase, and the facility will employ at least 35 full-time workers and dozens more part-time and contract employees.

NEXT: How Energy Efficient Are Facebook’s Data Centers?

The Facebook Data Center FAQ | Page Two | Page Three