A map of the global audience for Facebook, created by Paul Butler, visualizes the geographic spread of its user base. (Source: Facebook)

Facebook has invested more than $1 billion in the infrastructure that powers its social network, which now serves more than 845 million users a month around the globe. The company spent $606 million on servers, storage, network gear and data centers in 2011, and expects to spend another $500 million this year, Facebook revealed Wednesday in its filing for an initial public stock offering.

The company’s massive armada of servers and storage must work together seamlessly to deliver each Facebook page, the company said. “Loading a user’s home page typically requires accessing hundreds of servers, processing tens of thousands of individual pieces of data, and delivering the information selected in less than one second,” the company said.

Facebook reported in its SEC filing that it owns “network equipment” valued at $1.016 billion at the close of 2011. The number reflects the expense of rapidly building a massive Internet infrastructure, including Facebook’s shift from buying vendor gear and leasing data centers to building its own servers, racks and custom data centers.

Facebook: Investment is Paying Dividends

Although the shift requires huge capital investment up front, Facebook said its custom designs are “ significantly reducing our costs compared to the usage of traditional servers and leased data center facilities.”

But that investment is paying off. In 2011 Facebook made about $1 billion in profit on revenue of more than $3.7 billion.

Thus far, Facebook’s spending compares well with its Internet-scale peers. Google spent $951 million on its data center operations in just the fourth quarter of 2011, with infrastructure capital expenses of  $3.4 billion for all of 2011.

Facebook’s spending is likely to increase as it continues to build out its own data center infrastructure.  The company opened its first  data center in Prineville, Oregon in April 2011, which uses a simplified power distribution system and is filled with servers and racks customized for Facebook’s needs.

More Data Centers in the Pipeline

The social network currently has three more data centers under construction, including a second facility in Prineville and new data centers in Rutherford County, North Carolina and Lulea, Sweden.

The company’s construction costs are also competitive. Facebook spent about $210 million to build 28 megawatts of data center space in Prineville, which works out to about $7.5 million per megawatt. The most efficient providers are building scale-out data center space at between $5 million and $9 million per megawatt. Enterprise data centers, which require additional investment in on-site redundancy and security, can cost $15 million per megawatt.

Facebook said it expects to spend $180 million on real estate leases in 2012, but did not break out how much of that was dedicated to leasing of wholesale data center space, a market in which Facebook is one of the  largest tenants. We have previously estimated Facebook’s spending on data center leases to be at least $50 million a year.

One thing’s for sure: Facebook’s network is only getting bigger. “We plan to continue to significantly expand the size of our infrastructure, ” the company said. “We are investing in additional Facebook-owned data centers in the United States and Europe and we aim to deliver Facebook products rapidly and reliably to all users around the world.”

The interior of Facebook data center in Prineville, Oregon.

Facebook has invested $210 million to build the first phase of its new data center in Prineville, Oregon, which has a capacity of 28 megawatts of power, the company and local economic development officials revealed this week. The disclosures, released in an economic impact study and a community economic forum, were the first public confirmations of the cost and power usage of the Facebook project.

The new data, which shed light on the cost of operating Facebook’s massive server infrastructure, emerges as the company is said to be prepping papers for an initial public offering, which would include additional details about the company’s operations. The Oregon disclosures are part of Facebook’s effort to reinforce the benefits of its data center to the local economy, amid a dispute over property taxes and questions from some Prineville residents about the impact of data centers on the small community in central Oregon.

For Prineville, Facebook is a big business operation – a fact reflected in the power required to operate the first phase of the data center. The 28 megawatts of utility power for the 300,000 square foot first phase isn’t extraordinary for a data center of that size. But it stands out in Crook County, where all the homes and business other than Facebook use 30 megawatts of power.

Three Data Centers Possible

Facebook began building its Prineville facility in early 2010, and began operations in April 2011.  The company recently began building a second data center identical to the first, and its long-term plans for the campus include an option for a third 300,000 square foot facility. Based on the power requirements of the first building, that could translate into about 78 megawatts of electricity to support the campus.

Crook County Economic Development Manager Jason Carr, who discussed Facebook’s operations at a community forum last week, said the region will have no problem supporting that much demand from a single customer. Carr said the region has 720 megawatts of power available, with another 76 megawatts set to come online next year, and another 281 MW in mid-2014.

But that doesn’t mean that power is available in the right place at the right time. There are reports that Apple and at least one other large data center user have scouted locations in Prineville, as Facebook’s facility has attracted interest from other companies seeking to leverage the town’s ideal environment for using fresh air to cool servers.

“I will readily admit that probably some of you have been skeptical or asked questions about what good a data center does for Prineville,” Carr said at the economic fourm. “Why are they here? What are we doing with them?”

Economic Impact Study Released

The benefits of Facebook’s presence in Oregon are outlined in a new study from ECONorthwest, which asserts that the Prineville facility has generated $142.7 million in economic activity in the state of Oregon. Much of that economic impact is from the “spillover effect” of Facebook employees and construction workers spending money in the local economy. ECONorthwest has done a similar study on the impact of Intel’s operations in Oregon.

In terms of direct impact, the ECONorthwest study cites the following benefits:

• Of the $210 million to build and equip the data center, about $75.7 million of that capital spending going to Oregon companies and workers.
• Facebook employs 55 company or contract employees, with  total annual payroll of almost $3.3 million. The average annual salary of Facebook’s data center employees is $47,200, compared to a regional average of about $35,071 in 2010.
• Facebook’s capital spending created $24.4 million in economic activity, including $9.2 million in personal income and 234 jobs. These impacts include about 80 direct jobs for local construction and other workers employed on-site, and over $1.2 million in sales for local lodging, eating and drinking, and retail sectors as a result of per diem spending by non-local workers.

The study, which was commissioned by Facebook, arrives as the company is in a discussion with state and local officials about its potential tax bill. In August the company was told that the state was required to “assess any property, real and personal, tangible or intangible.” The company feared that the reference to “intangible” assets meant it could be taxed on the value of its brand.

The Associated Press reported last week that Oregon’s Department of Revenue will change its rules so data centers like Facebook’s are assessed by local tax authorities, not by the state. Oregon lawmakers are continuing to press for a new law to codify this solution, which would be useful in discussions with companies contemplating Oregon for new data centers.

One area where local officials praised Facebook was its approach to water use. Carr said Facebook has used only 28 gallons of water per minute thus far, while other existing industrial users in Crook County use between 60 and 173 gallons per minute.

The exterior of the Facebook data center in Prineville, Oregon. (Photo credit: Alan Brandt).

Does this story sound familiar? A Pacific Northwest state with affordable power and an ideal climate for free cooling attracts a cluster of data center projects. Then a large data center provider is told it may face a large, unanticipated tax bill due to an interpretation of state law. Legislators spring into action to draft bills to address the issue and maintain the state’s attractiveness as a data center destination?

No, it’s not Washington state, where this scenario played out in 2007-11. It’s neighboring Oregon, where a dispute with the state on property taxes affecting Facebook’s Prineville data center leaves the Oregon Department of Revenue searching for a fix. The Statesman Journal reports that the problem stems from how state officials originally defined Facebook as a company.  The tax issues began in August 2011 and in October tax authorities told the Bend Bulletin newspaper that Facebook faced an annual tax bill of $390,000, only to turn around the next day and say that the tax bill would be $26,000.

Facebook said it is concerned that the state is superseding the company’s local tax deal. With Facebook defined as a communication company, the Department of Revenue is working on a temporary administrative rule that would exempt certain communication companies from having their value assessed by the state if they don’t directly provide internet connectivity.

Seeking ‘Statutory Fix’

“The data center industry represents a great deal of potential for the state of Oregon at a time when the state has a great interest in diversifying the kind of industries and the kind of jobs here,” said Corey Owens, Facebook’s associate manager of public policy. Owens told lawmakers last Thursday that a ‘statutory fix’ would provide the certainty the company needs. The Senate Finance and Revenue Committee introduced a bill that would exempt property in enterprise zones from being assessed by the state.

Facebook has a few data center neighbors in Prineville who will surely be interested in the outcome of these tax discussions. Amazon recently deployed new data center capacity in Oregon and Apple may be considering a site in the area. Last summer Facebook said it would build a second data center on its Prineville campus.

What’s the lesson from Washington state? Lesson one is that these tax issues can have a significant impact on data center construction, which dried up in Quincy, Washington during the dispute. Lesson two is that the industry can mobilize to respond. In 2010 the Washington legislature passed an incentive package that resulted in a surge in building in Quincy, including expansions by Microsoft and Yahooo and new projects for Sabey Corp. and Dell.

Rich Miller contributed to this story.

Weihl left Google in November after six years at Google, during which he championed the company’s efforts to boost its use of renewable energy in its data centers. That included Power Purchase Agreements (PPAs) to add more wind power to utility grids that support Google data centers in Council Bluffs, Iowa and Pryor, Oklahoma. Google says it expects these two agreements alone to account for 15 percent of its company-wide energy usage by the end of 2012, pushing its overall renewable mix to 35 percent.

Another focus for Weihl was Google’s effort to limit its data center’s impact on the local water supply. Google has developed one of the industry’s most advanced programs to use recycled water in its data center cooling systems, and also works closely with municipalities to ensure that its operations don’t overwhelm the capacity of local sewer systems.

Last week Greenpeace and Facebook today said they will collaborate on the promotion of green energy sources and encourage major utilities to develop renewable energy generation, with Greenpeace also discontinuing a lengthy social media and PR campaign urging Facebook to “Unfriend Dirty Coal.” As part of the agreement, Facebook said it will seek to power its new data centers using clean and renewable energy. The company has already taken a major step in this direction in its latest data center project in Sweden, which will be powered primarily by renewable energy.

Rows of servers inside the new Facebook data center in Prineville, Oregon.

Facebook developed its Open Compute servers for its custom-built data centers. But the social network will soon be deploying those servers in some of its leased data center space as well.

Facebook has been working with landlord DuPont Fabros Technology (DFT) to implement its Open Compute designs in a data center in Ashburn, Virginia, according to Frank Frankovsky, Director of Hardware Design and Supply Chain at Facebook. The ability to run Facebook’s new hardware in leased facilities could be good news for the data center service providers, providing more flexibility as Facebook’s infrastructure makes the gradual transition to company-built facilities.

Frankovsky discussed Facebook’s infrastructure Monday in a keynote session at the Gartner Data Center Conference in Las Vegas, in which he reviewed steps enterprises could take to improve their data center efficiency without building their own custom infrastructure.

Progress on Open Compute in Multi-Tenant Sites

In an interview with Data Center Knowledge, Frankovsky discussed the Open Compute Project, an initiative to develop standardized hardware designs to democratize data center infrastructure. Frankovsky said there has been important progress in adapting Open Compute designs to multi-tenant data centers, a step that could make the energy efficient designs accessible to a broader range of users.

While it might be straightforward for companies to integrate the Open Compute designs when building their own data centers, Facebook’s customizations provide challenges in multi-tenant facilities, especially a streamlined power distribution with a decentralized UPS design. Facebook uses custom “triplet” racks that each house 30 of the 1.5U Facebook servers, with each row of racks enclosed in a hot-aisle containment system.

“There are ways to fit this technology into a non-customized data center,” said Frankovsky, noting Facebook’s collaboration with DuPont Fabros. “We’re handling the handoff to custom power at the PDU level.”

Facebook’s server power supplies include two connections, one for AC utility power and another for the DC-based UPS system. In the Ashburn installation, only the AC side of the design will be used.

Digital Realty Also Adapting Open Compute

Frankovsky said he’s also encouraged that Digital Realty Trust, another large data center operator, has adapted Open Compute concepts in reference designs for its data center pods. This makes it easier for customers interested in Open Compute designs to implement them in a new Digital Realty data center.

Facebook is a major customer of both wholesale providers, spending about $30 million a year on leases with Digital Realty Trust and $22 million a year on space with DuPont Fabros. Facebook has said that it will gradually shift its server capacity from leased data centers to company-owned facilities, migrating out of many third-party facilities as its leases expire.

The ability to easily implement Open Compute designs could make it easier for Facebook to adapt existing leased space to fit into its next-generation infrastructure. This could be important in markets like Ashburn, a key connectivity hub where Facebook is unlikely to build its own facility.

There’s just one problem.  ”We have no plans to build a data center in Taiwan,” said Facebook spokesman Michael Kirkland.

Could there be another huge project in Taiwan? Perhaps. Whenever there’s a large stealthy data center project, local chatter often attaches the project to the hot cloud builder of the moment. A few years ago, a number of projects were erroneously attributed to Google or Microsoft. Nowadays it’s Facebook or Twitter.

A dagram of the utility archotecture supporting the Facebook data center in lulea, Sweden, which will operate with fewer generators than Facebook's U.S. data centers.

For most data centers, backup generators serve as the last line of defense against downtime during utility power outages. Some major data centers use more than 30 diesel generators to ensure a ready supply of on-site generation.

Facebook is taking a different approach at its new data center in Sweden, reducing the number of backup generators by 70 percent. Facebook says the extraordinary reliability of the regional power grid serving the town of Lulea allows the company to use far fewer generators than in its U.S. facilities.

Using fewer generators reduces the data center’s impact on the local environment in several ways. It allows Facebook to store less diesel fuel on site, and reduces emissions from generator testing, which is usually conducted at least once a month.

Despite those benefits, Facebook’s approach would likely exceed the comfort level of many data center operators, who design their power systems to be as redundant as possible and see generators as a critical level of protection.

No Outages Since the 1970s

Local officials in Lulea say there has not been a single disruption in the area’s high voltage lines since 1979. The city lies along the Lulea River, which hosts several of Sweden’s largest hydro-electric power stations. The power plants along the river generate twice as much electric power as the Hoover Dam.

“There are so many hydro plants connected to the regional grid that generators are unneeded,” said Jay Park, Facebook’s Director of Datacenter Engineering. “One of the regional grids has multiple hydro power plants.”

Park says Facebook has configured its utility substations as a redundant “2N” system, with feeds from independent grids using different routes to the data center. One feed travels underground, while the other uses overhead utility poles.

Multiple Sources Support Regional Grid

While many data centers are supported by two power feeds, Facebook says the regional grid in Sweden is more resilient than the U.S. grid because it can tap power from so many sources -including  the national grid, other connected regional grids, and multiple hydro power plants. All of these sources would have to simultaneously fail before the Facebook data center would lose utility power.

Park says Facebook could probably operate entirely without generators in Lulea, but has taken the more cautious approach of installing a reduced number of generators. He emphasized that the reduction in generators is not a network-wide design change for Facebook, but limited to the Lulea facility.

“It’s very unique,” said Park. “We couldn’t do this in the U.S. because the areas we’re in don’t have multiple generation sources nearby.”

Yahoo: Data Centers With No Generators?

Facebook isn’t the first major Internet company to look at using fewer generators – or even none at all. Last year Yahoo said it was exploring scenarios in which it would build data centers without generators or UPS, and use its network to route around any power outages that occur at those facilities.

That’s a strategy that only the largest data center providers can contemplate, as it requires multiple data centers in major network capacity. Google has pursued a similar strategy during maintenance on some of its data centers, shifting capacity to other facilities (see How Google Routes Around Outages for details).

For more coverage of Facebook, see our Facebook Data Center FAQ or our Facebook Channel. For additional video, check out our DCK video archive and the Data Center Videos channel on YouTube.

An artist's illustration of what the new Facebook data center in Lulea, Sweden will look like when it is completed. (Click for larger image)

Facebook will build a huge new data center in northern Sweden to support the rapid global growth of its users, the company said today. The new data center in Lulea, Sweden will be Facebook’s first facility outside the United States. 

The location takes advantage of the cool climate in Lulea, a seaside town of 45,000 located at approximately the same latitude as Fairbanks, Alaska. That cool weather will allow Facebook to use outside air to cool the tens of thousands of servers that will occupy the new campus.

Facebook plans to build multiple data centers in Lulea, beginning with a 27,000 square meter (290,000 square foot) first phase that will be completed in late 2012 and begin supporting traffic in the first half of 2013.

Electricity from Green Power Sources

The Facebook Lulea data center will be powered primarily by renewable energy, a change from the social network’s first two company-built data centers in Oregon and North Carolina, which each relied upon utility power that originated primarily from coal. Facebook’s power sourcing had been highlighted by the environmental group Greenpeace, which launched a lengthy campaign urging Facebook to “Unfriend Dirty Coal.”

The expansion of Facebook’s infrastructure beyond the U.S. reflects the increasingly global makeup of its user base. More than 75 percent of Facebook’s 800 million users are located outside the United States. Building data centers closer to these users can improve the speed of their connection and overall Facebook experience.

“It’s the next step in our ongoing strategy of building our own infrastructure and moving away from leased facilities,” said Facebook spokesman Michael Kirkland. “We are expecting this data center to continue to help us reduce latency for our users in Europe and beyond.”

Sweden: Climate, Connectivity, Green Power

The Facebook announcement is being celebrated in Sweden, and particularly in Lulea, where economic development officials have been marketing the region as a data center destination due to its combination of a cool climate, strong connectivity and plentiful supply of cheap, renewable energy.

“The cool, dry climate will allow us to continue our practice of using outside air to cool our data centers,” said Kirkland. The average daily temperature in Lulea ranges from high of 41 degrees F (5 degrees C) to low of 27 degrees F (-2.5 degrees C). The area averages just four days a year with high temperatures exceeding 25°C (77 degrees F).

Sweden is ranked second in the world for its telecommunications infrastructure by the International Telecommunication Union (ITU), trailing only South Korea. Lulea has connectivity from five carriers: backbone provider TeliaSonera and Tele 2, TDC, Telenor and Banverket.

Energy Sourcing an “Important Consideration”

The nearby Lule River produces about 13.6 million megawatt hours of hydro-electric power, equal to 10 percent of Sweden’s total demand for electricity. Officials in Lulea say the area has some of the cheapest power rates in all of Europe.

“This will be the first Facebook data center powered primarily by renewable power, primarily hydro,” said Kirkland. “It’s a really important consioderation for us. Obviously, it’s not the only consideration.”

“This is a great step forward for Facebook, but we would like more details on how much renewable energy will power its data centre in Luleå,” said Casey Harrell, Greenpeace IT analyst. “With the IT sector one of the fastest growing consumers of electricity in the world, Facebook’s taking leadership on renewable energy could help determine whether we have a dirty ‘cloud’ or not.”

Facebook says the Lulea facility with be an energy-efficient data center, and employ many of the techniques used to conserve power at its Prineville site. The Oregon facility uses evaporative cooling instead of a chiller system, continuing a trend towards chiller-less data centers and water conservation. Facebook also says it will use waste heat from servers to heat office areas.

Inside the building, Facebook is expected to implement the server and data center designs outlined in the Open Compute Project, which the company launched in February to release its custom designs for servers, power supplies and UPS units.

We’ll have more details on the new facility after today’s press conference in Sweden and the Facebook Open Compute Summit in New York.

An aerial view of the huge new Facebook data center under construction in Rutherford County, North Carolina. Click to see a larger version. (Credit: Facebook).

Facebook has begun construction on a second huge data center in Forest City, North Carolina, even as it prepares to bring its first facility online. Facebook Data Center Manager George Henry said Monday that the second building will mirror Building 1, a 300,000 square foot structure, which is now physically complete and being populated with its first batch of servers.

When Facebook first announced its Rutherford County facility last November it had 500 million users. Today, just 11 months later, Facebook has more than 800 million users. The company has also begun work on a second facility at its data center campus in Prineville, Oregon.

Room for More Servers
Facebook purchased 150 acres of land from Rutherford County for the $450 million project. The company built its first data center on vacant land, but also demolished a former Mako Marine factory located on the site to clear a second site.

The expansion will mean additional work for Facebook’s construction team of DPR Construction and Fortis Construction Inc.  Approximately 500 people per day have worked at the Facebook site thus far. The announcement of the second phase ensures the same type of activity for at least another full year.

“We believe the expanding operations and continued construction activities will have a positive impact on Rutherford County’s economy,” Henry told county commissioners Monday.

Facebook has hired about 30 full-time employees at the data center, and expects a small increase of approximately 10 jobs at the Rutherford campus as a result of this expansion.

Facebook says the North Carolina facility with be an energy-efficient data center, and employ many of the techniques used to conserve power at its Prineville site. The Oregon facility uses evaporative cooling instead of a chiller system, continuing a trend towards chiller-less data centers and water conservation. The North Carolina site is in a warmer climate, which will reduce the scope of free cooling. But the facility will likely mirror Prineville’s practice of re-using excess heat expelled by servers to heat office space in the building.

Inside the building, Facebook is expected to implement the server and data center designs outlined in the Open Compute Project, which the company launched in February to release its custom designs for servers, power supplies and UPS units.