Facebook: Managing Epic Growth in Real-Time

Facebook VP of Technical Operations Jonathan Heiliger speaks at the O'Reilly Velocity 2009 conference in San Jose, Calif.

The largest Internet sites manage a precarious balance between innovation and reliability. That’s a particular challenge for Facebook, which is experiencing epic user growth while rolling out new features on a regular basis. How does it manage it?

“We fail all the time,” said Jonathan Heiliger, the VP of Technical Operations for Facebook. “It’s not without danger. Our goal is to make (failure) transparent to our users, and create an environment where it’s safe for our employees to fail.”

Heiliger was the keynote speaker at the O’Reilly Velocity 2009 conference Tuesday in San Jose, where he  discussed the challenges presented by Facebook’s growth.  “We believe the most effective technical organizations are those that can change fast, and that takes teamwork,” said Heiliger.

At Facebook, that has meant cultivating a culture of collaboration between engineering and operations,  two groups that often are in conflict. While engineers are eager to innovate, operations covets stability and uptime. “You get conflict,” said Heiliger. “I think in every company there’s conflicts between operations and engineering.”

To bridge the gap, Facebook changed the process for introducing new features. In most organizations, the engineering team writes code, which is then tested by a quality assurance (QA) team before being deployed and becoming the responsibility of the operations team. Heiliger pursued a different approach.

“We don’t actually have QA,” he said. “At Facebook, every engineer is responsible for the cradle-to-grave lifecycle of their code and their application. You want to put engineering as close to the customer as possible.”

Facebook Chat: 1 Billion Messages A Day

Activity on Facebook Chat has reached 1 billion messages per day, the company said on its Facebook Engineering blog. It’s the latest in a series of huge numbers and milestones for the fast-growing social networking service. Earlier Monday ComScore reported that Facebook had officially passed MySpace as the largest social network among U.S. users, the last demographic for which MySpace could still claim dominion.

As with many earlier engineering posts, Facebook’s Chris Piro cited the company’s constant performance tweaking for the Chat service. “We’ve invested a lot in making Chat stable and scalable in the past, and we continue making improvements even now,” writes Piro. “We’ve increased the capacity of our load balancers so we can accept more concurrent users, and we’re investigating ways to make the backend and frontend more robust against failures.”

Facebook: $20 Million a Year on Data Centers

A look at the fully-packed racks inside a Facebook data center facility.

Facebook appears to be spending $20 million to $25 million a year for the data center space that houses its servers, according to an analysis of the company’s data center infrastructure. The company’s costs will rise later this year, when it adds a new data center in Virginia. While that’s a lot of money, it’s far less than Microsoft or Google spend building their data centers, and less than many enterprise companies spend on facilities.

Facebook’s extraordinary growth has forced the company to invest in rapidly expanding its infrastructure. The social network recently crossed the 200 million user barrier, prompting reports that Facebook must raise new capital to pay for servers and data centers. Other sources say the estimates of Facebook’s burn rate are overstated, and the company has enough cash to operate for several years.

As its growth has accelerated in the past two years, Facebook has managed its infrastructure costs through its relationships with the two largest “wholesale” data center landlords, the real estate investment trusts Digital Realty Trust and DuPont Fabros Technologies. Here’s what we know about Facebook’s spending on its major data center commitments:

• Facebook is paying $10.9 million a year for 114,168 square feet of space in two Silicon Valley data centers it is leasing from Digital Realty (DLR), according to data from the landlord’s quarterly report to investors.
• The social network is also leasing data center space in Ashburn, Virginia from DuPont Fabros (DFT). Although the landlord has not published the details of Facebook’s lease, Rackspace (RAX) recently said in an SEC filing that it is paying about $5 million a year for a similar amount of space  in the same Ashburn data center used by Facebook (known as ACC4).
• Facebook also hosts equipment in a Santa Clara, Calif.  data center operated by Terremark Worldwide (TMRK), a Palo Alto, Calif. facility operated by Switch & Data (SDXC) and at least one European data center operated by Telecity. These are believed to be substantially smaller footprints than the company’s leases with Digital Realty and DuPont Fabros.

That adds up to an estimated $16 million for the leases with the two data center REITs. When you add in the cost of space for housing equipment at Terremark, Switch and Data, Telecity and other peering arrangements to distribute content, we arrive at an estimate of between $20 million and $25 million in annual data center costs for Facebook.

A Look Inside Facebook’s Data Center

If your users are uploading 40 million photos a day, what does your data center look like? A new video seeking to recuit engineers for Facebook provides a glimpse of one of the social network’s data center facilities, along with some facts about the social network’s amazing growth. The company’s data centers store more than 40 billion photos, and users upload 40 million new photos each day - about 2,000 photos every second. Not surprisingly, the racks are packed. The facility is using a raised-floor design, with no containment but generous spacing between racks. Here’s the video, which runs about 2 minutes, 30 seconds.

For more about Facebook’s data centers, check out some of our previous coverage:

• Facebook Expanding Its Data Centers, Again
• Facebook Pushes Limits for memcached
• The Backstory: Facebook’s Virginia Data Center
• Facebook Now Running 10,000 Servers

If you’re interested in opportunities for engineers at Facebook, see their careers page.

Facebook Expanding its Data Centers. Again

Facebook has recently signed leases for additional data center space as it expands its infrastructure to support the social network’s phenomenal growth. Facebook, which has just surpassed 200 million users, is also reportedly seeking up to $100 million for additional servers.

In recent weeks Facebook has reserved a large chunk of space in the huge ACC5 data center facilty being built in Ashburn, Virginia by DuPont Fabros Technology (DFT). Facebook is a tenant in DuPont Fabros’ ACC4 data center in Ashburn, where it leases about 10,000 square feet of space to house its servers. The company’s lease in ACC5 is at least double that size, and will use more than 6 megawatts of power.

DuPont Fabros announced the large lease at ACC5, but has not disclosed the tenant. However, other sources have named Facebook as the customer.  

Facebook has also signed a lease for additional data center space from Digital Realty Trust. The new lease, signed in the fourth quarter of last year, is the company’s second with Digital Realty, the world’s largest data center landlord. Back in February 2008, Facebook leased an entire data center facility in Santa Clara.

Facebook Adding 600,000 Users A Day

Facebook said this week that it has reached 140 million users, and its growth is accelerating. The social network released updated data this week that reveals why the Facebook engineering team has been working so hard to scale the site’s infrastructure.

Inside Facebook has been tracking the statistics released by Facebook, and says that in recent weeks the site’s growth has accelerated to more than 600,000 new users per day, up from about 300,000 to 400,000 a day. “If Facebook continues at this rate, it could add up to 20 million new users in December and reach 200 million active users by March,” Justin Smith writes.

Facebook Pushes Limits for Memcached

When engineers discuss massive web scalability, the conversation often turns to memcached, the object caching system used to speed database-driven dynamic web application. Facebook has been among the sites that has leveraged the capabilities of memcached to keep pace with rapid audience growth.

“Here at Facebook, we’re likely the world’s largest user of memcached,” writes Paul Saab of the Facebook engineering team in a post that details how Facebook has refined its use of memcached to handle up to 200,000 UDP requests per second in production. “The total throughput achieved is 300,000 UDP requests/s, but the latency at that request rate is too high to be useful in our system,” Saab writes. “This is an amazing increase from 50,000 UDP requests/s using the stock version of Linux and memcached.”

See the Facebook Engineering Blog for details on how they’ve done it, and High Scalability for additional analysis.

Facebook said it hopes to get its enhancements incluced in the official memcached code, but in the meantime has published its changes to memcached on the social code repository github.

Facebook, Twitter End Talks on $500M deal

Kara Swisher at AllThingsD reports top executives of Facebook and Twitter recently “ended several weeks of serious talks, in which Facebook was offering to acquire Twitter for $500 million of its stock.” There had been some chatter about a deal, but talks were apparently more serious than was widely known.

The Twitter team apparently decided $500 million wasn’t a rich enough price. Since the company has still not developed a clear plan for making money from its popular microblogging service.

There was also a cost issue for Facebook. If Twitter was offered to Facebook’s 120 million users, Facebook execs estimated that it might have to deal with huge SMS fees–up to $75 million annually.“Facebook has its own revenue-generating challenges,” said one person close to the company. “As much as Twitter would give them a lift in the status area, it was still a worry.”