Facebook Now Has 30,000 Servers

A look at the fully-packed racks inside a Facebook data center facility.

How many servers does Facebook have? For some time now, the stock answer has been “more than 10,000 servers,” a number the company began using in April 2008. Facebook has continued to use that number, even as it has soared past 300 million users and dramatically expanded its data center space.

We now have an update: Facebook has 30,000 servers supporting its operations. That number comes from Jeff Rothschild, the vice president of technology at Facebook, who discussed the company’s infrastructure in a presentation last week at UC San Diego (link via High Scalability).

“Today we have somewhere in the neighborhood of 30,000 servers,” Rothschild said during the Q&A session following his talk, adding that the number ”will be different today than it was yesterday” because Facebook is adding capacity on a daily basis.

Facebook, Google Ready for Faster Ethernet

As Facebook was announcing that it has reached 300 million users last week, network engineer Donn Lee was making the case for faster Ethernet networks – definitely 100-Gigabit Ethernet, and perhaps even Terabit Ethernet.

It is “not unreasonable to think Facebook will need its data center backbone fabric to grow to 64 terabits a second total capacity by the end of next year,” Lee told attendees of a seminar hosted by the Ethernet Alliance, an industry consortium focused on the expansion of Ethernet technology.  

10 Gigabit Ethernet is currently the fastest rate available, although higher capacity can be achieved by aggregating 10 Gigabit links. The Ethernet Alliance event last week brought together networking developers working on the IEEE 802.3ba standard for 40/100 Gigabit Ethernet. Presentations from Facebook’s Lee and Google network architect Bikash Koley offered a deep dive into network technologies, architecture and the future of Ethernet standards. The next phase of development envisioned is Terabit Ethernet, although some stakeholders see 400 Gigabit Ethernet as a likely interim step.

How Facebook Serves Up 300 Million Users

A look at the fully-packed racks inside a Facebook data center facility.

Facebook announced last night that it now has more than 300 million active users, meaning the social network has been adding users at an average rate of 670,000 per day. Most of those users are uploading photos and videos, which requires a lot of storage and computing assets.

That’s why Facebook has been aggressively expanding its infrastructure by leasing wholesale data center space from DuPont Fabros Technology (DFT)  in Virginia and Digital Realty Trust in California. Here’s a look at some of our coverage of the growth of Facebook’s data center infrastructure:     

• Facebook Makes Big Investment in Data Centers: The company has just locked down a large chunk of data center space in northern Virginia that will provide room for thousands of additional servers for growth in 2011 and beyond.
• Facebook: Managing Epic Growth in Real-Time: CTO Jonathan Heiliger presents an overview of Facebook’s back-end operations at the Velocity 2009 conference in June.
• Facebook Spending $20 Million a Year on Data Centers: An analysis of Facebook’s data center leases as of  May 2009.
• A Look Inside Facebook’s Data Center: A recruitment video provides a glimpse inside the server-packed racks and aisles of a Facebook data center. 
• Facebook Expanding Its Data Centers, Again: In early 2009 the company leased additional space in  its East Coast hub in Ashburn, Virginia.
• Facebook Pushes Limits on Memcached: Caching is key to massive web scalabilty. Here’s how Facebook is extending a popular caching technology.
• Facebook Borrows $100 Million to Buy Servers: Last year Facebook borrowed money to finance additional investment in servers to support its rapid growth.  

Facebook Makes Big Investment in Data Centers

The exterior of the DuPont Fabros Technology ACC5 data center in Ashburn, Va. during construction earlier this year. Facebook has pre-leased additional space in the facility.

Facebook has made a major commitment to its future infrastructure, locking down a large chunk of data center space in northern Virginia that will provide room for thousands of additional servers to power its fast-growing social networking hub. The lease in Ashburn, Va., which doesn’t start until 2011, represents a substantial financial commitment – as much as $125 million over the life of the lease, by some analysts’ math.   

Facebook has pre-leased 33,000 square feet of additional space in a huge data center being built by DuPont Fabros Technology in Ashburn, Virginia known as ACC5. Facebook already operates a data center in an adjacent building known as ACC4, and is about to move into about 30,000 square feet of space it has leased in the first phase of ACC5, which has just opened to tenants.

Planning Ahead for Major Growth
The new lease is in Phase II of ACC5, which has not yet been completed, and provides Facebook with 6.8 megawatts of power to support its IT gear. The lease doesn’t begin until January of 2011, meaning Facebook is planning ahead and expecting to fill the space in its first two Virginia data centers within the next 15 months. Facebook, which won’t begin making payments until the lease commences, wouldn’t comment on the length or total outlay involved in the agreement.

“We do not disclose specifics with regard to our lease financials, but we can confirm that we have signed an agreement,” Facebook said in a statement. “This agreement continues with our effort to add data center capacity to support our growing business.”

Just In Case: Facebook’s “Nuclear Option”

The Facebook infrastructure team has blogged about the challenges involved in its recent launch of usernames, which allow Facebook users to personalize or brand their profile URL. With more than 200 million users and a single login page URL, the launch of usernames was “unique in its preparation and potential for trouble,” Facebook’s Tom Cook writes on the Facebook engineering blog.

Facebook VP of Technical ops Jonathan Heiliger discussed some of the launch management strategies at last week’s Velocity 2009 and Structure 09 conferences, including the use of a “dark launch” and the decision to go live with the username feature at a period of low traffic.

But Cook digs into some interesting “just in case” scenarios the Facebook team developed to manage a variety of traffic and web site load scenarios. “Facebook comprises hundreds of interlocking systems, although to users it’s presented as a simple web page,” he writes. “Throttling back the behavior of certain facets allows us to lighten the demand on our infrastructure without compromising major site functionality.”

Facebook: Managing Epic Growth in Real-Time

Facebook VP of Technical Operations Jonathan Heiliger speaks at the O'Reilly Velocity 2009 conference in San Jose, Calif.

The largest Internet sites manage a precarious balance between innovation and reliability. That’s a particular challenge for Facebook, which is experiencing epic user growth while rolling out new features on a regular basis. How does it manage it?

“We fail all the time,” said Jonathan Heiliger, the VP of Technical Operations for Facebook. “It’s not without danger. Our goal is to make (failure) transparent to our users, and create an environment where it’s safe for our employees to fail.”

Heiliger was the keynote speaker at the O’Reilly Velocity 2009 conference Tuesday in San Jose, where he  discussed the challenges presented by Facebook’s growth.  “We believe the most effective technical organizations are those that can change fast, and that takes teamwork,” said Heiliger.

At Facebook, that has meant cultivating a culture of collaboration between engineering and operations,  two groups that often are in conflict. While engineers are eager to innovate, operations covets stability and uptime. “You get conflict,” said Heiliger. “I think in every company there’s conflicts between operations and engineering.”

To bridge the gap, Facebook changed the process for introducing new features. In most organizations, the engineering team writes code, which is then tested by a quality assurance (QA) team before being deployed and becoming the responsibility of the operations team. Heiliger pursued a different approach.

“We don’t actually have QA,” he said. “At Facebook, every engineer is responsible for the cradle-to-grave lifecycle of their code and their application. You want to put engineering as close to the customer as possible.”

Facebook Chat: 1 Billion Messages A Day

Activity on Facebook Chat has reached 1 billion messages per day, the company said on its Facebook Engineering blog. It’s the latest in a series of huge numbers and milestones for the fast-growing social networking service. Earlier Monday ComScore reported that Facebook had officially passed MySpace as the largest social network among U.S. users, the last demographic for which MySpace could still claim dominion.

As with many earlier engineering posts, Facebook’s Chris Piro cited the company’s constant performance tweaking for the Chat service. “We’ve invested a lot in making Chat stable and scalable in the past, and we continue making improvements even now,” writes Piro. “We’ve increased the capacity of our load balancers so we can accept more concurrent users, and we’re investigating ways to make the backend and frontend more robust against failures.”

Facebook: $20 Million a Year on Data Centers

A look at the fully-packed racks inside a Facebook data center facility.

Facebook appears to be spending $20 million to $25 million a year for the data center space that houses its servers, according to an analysis of the company’s data center infrastructure. The company’s costs will rise later this year, when it adds a new data center in Virginia. While that’s a lot of money, it’s far less than Microsoft or Google spend building their data centers, and less than many enterprise companies spend on facilities.

Facebook’s extraordinary growth has forced the company to invest in rapidly expanding its infrastructure. The social network recently crossed the 200 million user barrier, prompting reports that Facebook must raise new capital to pay for servers and data centers. Other sources say the estimates of Facebook’s burn rate are overstated, and the company has enough cash to operate for several years.

As its growth has accelerated in the past two years, Facebook has managed its infrastructure costs through its relationships with the two largest “wholesale” data center landlords, the real estate investment trusts Digital Realty Trust and DuPont Fabros Technologies. Here’s what we know about Facebook’s spending on its major data center commitments:

• Facebook is paying $10.9 million a year for 114,168 square feet of space in two Silicon Valley data centers it is leasing from Digital Realty (DLR), according to data from the landlord’s quarterly report to investors.
• The social network is also leasing data center space in Ashburn, Virginia from DuPont Fabros (DFT). Although the landlord has not published the details of Facebook’s lease, Rackspace (RAX) recently said in an SEC filing that it is paying about $5 million a year for a similar amount of space  in the same Ashburn data center used by Facebook (known as ACC4).
• Facebook also hosts equipment in a Santa Clara, Calif.  data center operated by Terremark Worldwide (TMRK), a Palo Alto, Calif. facility operated by Switch & Data (SDXC) and at least one European data center operated by Telecity. These are believed to be substantially smaller footprints than the company’s leases with Digital Realty and DuPont Fabros.

That adds up to an estimated $16 million for the leases with the two data center REITs. When you add in the cost of space for housing equipment at Terremark, Switch and Data, Telecity and other peering arrangements to distribute content, we arrive at an estimate of between $20 million and $25 million in annual data center costs for Facebook.

A Look Inside Facebook’s Data Center

If your users are uploading 40 million photos a day, what does your data center look like? A new video seeking to recuit engineers for Facebook provides a glimpse of one of the social network’s data center facilities, along with some facts about the social network’s amazing growth. The company’s data centers store more than 40 billion photos, and users upload 40 million new photos each day – about 2,000 photos every second. Not surprisingly, the racks are packed. The facility is using a raised-floor design, with no containment but generous spacing between racks. Here’s the video, which runs about 2 minutes, 30 seconds.

For more about Facebook’s data centers, check out some of our previous coverage:

• Facebook Expanding Its Data Centers, Again
• Facebook Pushes Limits for memcached
• The Backstory: Facebook’s Virginia Data Center
• Facebook Now Running 10,000 Servers

If you’re interested in opportunities for engineers at Facebook, see their careers page.