A photo of the Google data center in Goose Creek South Carolina.

Google has declined a jobs-related tax break from the state of South Carolina, and will not claim a $2.5 million grant that required that the company create 200 new jobs in the state, according to local media. The incentives were part of the package Google negotiated with state officials in the site selection process for its data center in Goose Creek, S.C.

The news provides the latest cautionary tale about the use of tax incentives to lure huge data center projects. These projects bring huge investment in the local economy, but don’t always deliver on the level of job creation envisioned by the incentive packages – which in the case of data centers, aren’t enormous to begin with. 

Google previously informed the state of North Carolina that it won’t meet the job creation criteria for a $4.7 million state grant for its data center project in Lenoir. The grant required the company to create 200 jobs in four years, but Google has slowed the pace of construction in Lenoir as part of a broader initiative to manage capital expenditures during the economic slowdown.

Google also expected to create 200 jobs in Goose Creek, but apparently has not met that number. “Given the current economy and the difficulty in forecasting our business climate, we do not believe that the (job tax credits) would be a wise investment for either Google or South Carolina at this time,” Google spokeswoman Emily Wood told the State newspaper.

The newspaper reports that Google has completed one data center at its Goose Creek site and built the shell for a second data center, but apparently has not installed any equipment. Wood said the company doesn’t have a time frame for opening the second data center, and that company officials will make that call on the basis of Google’s need for more data-storage capacity.

Photo of Google’s Goose Creek data center from The Digitel via Flickr.

Most of Google’s recent high-profile outages have been caused by routing or network capacity problems, including outages in May and September of last year (see How Google Routes Around Outages for more). But not so with the Feb. 24 event.

“The underlying cause of the outage was a power failure in our primary datacenter,” Google reported. “While the Google App Engine infrastructure is designed to quickly recover from these sort of failures, this type of rare problem, combined with internal procedural issues extended the time required to restore the service.”

Power Down for 30 Minutes
Data center power outages typically fall into two categories: those in which the entire data center loses power for an extended period, and those in which power is restored relatively quickly but hardware within the data center has trouble restarting properly. The Google App Engine downtime appears to fall into the latter category. Power to the primary data center was restored within a half hour, but a key group of servers failed to restart properly. The somewhat unusual pattern of the recovery presented the first challenge.

“We failed to plan for the case of a power outage that might affect some, but not all, of our machines in a datacenter (in this case, about 25%),” Google reported. “In particular, this led to incorrect analysis of the serving state of the failed datacenter and when it might recover.”

This in turn complicated the decision-making process about whether and when to shift Google App Engine to a second “failover” facility. It’s also the point where Google’s documentation became part of the story.

Dated Documentation
“Recent work to migrate the datastore for better multihoming changed and improved the procedure for handling these failures significantly,” Google noted. “However, some documentation detailing the procedure to support the datastore during failover incorrectly referred to the old configuration. This led to confusion during the event.

“Although we had procedures ready for this sort of outage, the oncall staff was unfamiliar with them and had not trained sufficiently with the specific recovery procedure for this type of failure,” the incident report continued.

As a result, Google engineers had trouble deciding whether to commit to the primary or secondary data center at a key moment. At one point, they reversed a decision to focus on the failover data center, believing the primary facility might have fully recovered. It hadn’t, and the misstep resulted in a slightly longer outage for App Engine customers.

Lessons Learned, Steps Outlined
As in any good post-mortem, the Google team shared a series of steps it is taking to address the issues that arose in the Feb. 24 incident:

• Google will schedule additional drills by all oncall staff to review production procedures, including those for “rare and complicated procedures.” All members of the team will be required to complete the drills before joining the oncall rotation.
• The company will also implement a regular bi-monthly audit of operations docs, and ensure that all out-of-date docs are properly marked “Deprecated.”
• The company will “establish a clear policy framework to assist oncall staff to quickly and decisively make decisions about taking intrusive, user-facing actions during failures. This will allow them to act confidently and without delay in emergency situations.”
• Google said it will make a major infrastructural change in App Engine, which currently provides a “one-size-fits-all” Datastore. In
  the wake of the Feb. 24 outage, Google says it will offer two different Datastore configurations: the current option of low-latency and lower availability during unexpected failures, and a new option for higher availability using synchronous replication
  for reads and writes, “at the cost of significantly higher latency.”

Read the entire post-mortem report for additional details, including a full timeline of the incident.

The scope and detail of the report drew plaudits. Lenny Rachitsky of Transparent Uptime blog called it “nearly a perfect model for others.”

“A vast majority of the issues were training related,” Rachitsky wrote. “This is an important lesson: all of the technology and process in the world won’t help you if your on-call team is unaware of what to do. This is especially true during the stress of a large incident.”

It turns out that’s not quite correct. “These fuel cells aren’t powering any off-site data centers,” said a Google spokesperson. “Instead, Bloom fuel cells are powering a portion of Google’s energy needs at our headquarters right here in Mountain View. This is another on-site renewable energy source that we’re exploring to help power our facilities. We have a 400kW installation on Google’s main campus. Over the first 18 months the project has had 98% availability and delivered 3.8 million kWh of electricity.”

The Bloom Energy units run on methane or other hydrocarbons. The machine produces electricity, as well as some heat, carbon dioxide and water. While 400 kilowatts is a lot of power for some commercial buildings, it’s a fraction of what would be needed for a major data center. The same goes for the 98 percent availability, as data centers typically shoot for at least “four nines” (99.99 percent uptime) and beyond.

Despite those issues, there are a number of data center projects that have incorporated fuel cells using natural gas or biogas. Some previous examples:

• T-Systems is using a “hot module” fuel cell to provide power for a server room in a facility in Munich, Germany, which runs on biogas supplied by a planet in nearby Pliening.
• Fujitsu has used a fuel cell in its Sunnyvale data center. The fuel cell produce 200 kilowatts of power, which is enough to power half of the cooling needed in Fujitsu’s data center.   
• A Syracuse University data center is susing gas-powered microturbines, which generate electricity, while the hot exhaust is piped to the chiller room, where it is used to generate cooling for the servers and both heat and cooling for an adjacent office building.
• Verizon has been using fuel cell technology to power one of its facilities in Garden City, N.Y., on Long Island. Seven fuel cells generate power for a 292,000-square-foot facility that provides telephone and data services to some 35,000 customers on Long Island

The primary barrier to use of fuel cells in data centers has been the up-front cost of the units. 60 Minutes reports that each Bloom unit costs $700,000 to $800,000. Venture capitalist John Doerr of Kleiner Perkins Caufield & Byers, one of Bloom Energy’s backers, said the Bloom Box is intended to replace the grid for its customers. “It’s cheaper than the grid, (and) it’s cleaner than the grid,” Doerr told 60 Minutes.

We’ll no doubt hear more in a Wednesday press conference to officially launch Bloom Energy.

The Wikimedia Foundation, the non-profit that runs Wikipedia, houses its entire infrastructure on about 350 servers in data centers in Florida and Amsterdam. The grant from Google, which can spend more than half a billion dollars on a single data center, will help beef up Wikipedia’s back end.

Wikimedia’s spokesman Jay Walsh said that expanding the foundation’s data centers will be the first task facing new CTO Danese Cooper. “We expand incrementally as required to keep up with increased traffic demands, but we’re looking at a more broad, long-term solution for growing to keep meeting those demands,” Walsh told DataCenterDynamics.

The Wikipedia site is hosted on about 300 servers in a primary data center in Tampa, Fla., and another 50 servers at the EvoSwitch data center in Amsterdam that provide content storage and delivery.

An architectural rendering of the new Facebook data center planned for Prineville, Oregon.

Facebook has responded to growing criticism of its power choices for its new data center in Prineville, Oregon. This is one of the first cases in which a data center’s energy sourcing has attracted this kind of public attention, but it won’t be the last.  

Earlier this month we noted a report that Facebook’s new Oregon data center, which has been designed to be highly energy-efficient, would be getting its power from a local utility that uses coal to generate the majority of its power. This news, initially reported by SearchDataCenter, has been getting attention from environmental groups and green blogs. The issue was highlighted on TreeHugger and Change.org, and has even led to the creation of a Facebook group (Tell Facebook to use Clean Energy for its Data Center).

Facebook’s Response 
Yesterday Facebook responded via our comment section on the original post, with a statement from spokesperson Lee Weinstein. Since Facebook has taken its lumps on this issue, I felt its response should get equal visibility. Here’s Weinstein’s comment:

I’m writing on behalf of Facebook to share their response to the issues you’ve posed. Most electrical commercial and residential power in the United States comes from a variety of sources. Our new data center will be receiving our power through PacifiCorp, which like most utilities has a diverse generation portfolio including hydro, geothermal, wind and coal. PacifiCorp is now the #1 utility owner operator of renewables, having grown their portfolio 2,400 percent over the past three years.

When it comes online in early 2011, the new Facebook data center will also be one of the most energy efficient in the world, featuring an innovative cooling system created for the unique climate characteristics in Prineville, Oregon.

The new, world class energy-efficiency technologies the Facebook data center will utilize include an evaporative cooling system; an airside economizer that will bring colder air in from the outside; re-use of server heat to warm office space in the colder months; and new patent pending highly efficient electrical design will reduce electricity usage by up to 12 percent. The entire facility will be built to LEED Gold standards.

The State of Oregon has a very aggressive Renewable Portfolio Standard, calling for 25 percent of power in the state to be produced by renewable resources by 2025. Facebook believes this policy will ensure continued growth of renewable generation resources. Facebook’s commitment is, regardless of generation source, to use electricity as wisely and as efficiently as possible.”

I’ll circle back to our commentary on the original post: “This is a good example of the ‘clean vs. cheap’ dilemma faced by data centers with massive power requirements. Companies like Google and Facebook want to be as green as possible, but must also control costs. The economics of on-site generation of using wind or solar power still don’t add up for most providers. That means the best way to use more renewable energy in the data center is to buy it from the utility company.” Sometimes that’s not as easy as you’d think, even in Oregon.

The irony is that companies like Facebook, which make energy efficiency a priority and are open about their interest in sustainability, are held to a higher standard when it comes to energy sourcing. We’ve previously noted Facebook’s focus on efficiency in the leasing and design of its data centers:

• Facebook Goes Green With New Data Center Space: The breakout growth of Facebook is turning out to be an economic stimulus plan for data center landlords. And the fast-growing social network’s infrastructure isn’t just getting bigger, it’s getting greener, with a focus on LEED Gold or Platinum facilities.
• Facebook Follows Google to Data Center Savings: Facebook says it’s streamlining its servers, and also plans to adopt a novel power distribution design pioneered by Google.
• It’s Official: Facebook is Oregon’s ‘Company X’: Facebook says the 147,000 square foot Prineville data center will be designed to LEED Gold standards and is expected to have a Power Usage Effectiveness (PUE) rating of 1.15.

Google to Face Similar Scrutiny
Another company bearing the weight of high expectations is Google, whose actions are often viewed through the prism of the ”don’t be evil” sentiment expressed by its founders. The company’s sourcing of power for its data centers is likely to come under similar scrutiny. Major media outlets are increasingly interested in Google’s energy use in its data centers, including how much it’s using and how “green” that power is.

Google is open about its interest in both data center efficiency and sustainability, and its foundation has made significant investments in wind, solar and geothermal power. But the company hasn’t been forthcoming about how much electricity its data centers use, and the media’s efforts to fill in the blanks can easily go awry. A recent example: the wild-ass guess by The Wall Street Journal that Google’s energy use was “roughly equivalent to the output of two large conventional power plants.”

Google has made enormous investments in building what it believes are the world’s most efficient data centers. Other media outlets are taking a hard look at Google’s energy use, and whether it is aligned with its commitment to sustainability. Will Google wait to let others characterize its data center energy use, or disclose more about its current power usage and sourcing and the company’s plans to make it more sustainable over time?

Google data center in Australia? Speculation over the possibility of Google (GOOG) building a data center in Australia has picked up again.  ZDNet Australia reports that there is “intense interest” in Google’s Application suite and quoted Google’s Deepak Ramanathan, head of Asia-Pacific market development as saying that the “search giant was not ruling out an Australian data centre for services like Gmail, and that discussions were ongoing.”  Global IT outsourcer CSC is pushing into cloud computing opportunities in Australia with Microsoft products and a recent strong interest in Google Apps from Australian enterprises and public sector organizations. ZDNet Australia also recently reported on IT services company Infosys opening an $8 million Melbourne data centre.

VISI expands Minnesota data center. Data center services and hosting company VISI announced initiation of its demand-driven expansion of their St. Paul data center.  The 7,000 square foot expansion project will bring the downtown St. Paul facility to 18,000 net square feet when completed.

Active Power reports 2009 results. Critical power systems manufacturer Active Power (ACPW) announced results for its fourth quarter and fiscal year ending December 31, 2009.  Active Power received more than $8 million in PowerHouse orders in the second half of 2009, exceeding all of the company’s containerized sales in 2008.  They also shipped 86 flywheel based UPS systems at an average selling price of $90,000 per unit during the fourth quarter of 2009.  Fourth quarter revenue increased 64 percent to $14 million and gross profit margin was 19 percent.

Level 3 expands Florida presence.  Level 3 (LVLT) announced it has expanded operations and is enhancing its local presence in Central Florida, including the Tampa-St. Petersburg and Orlando metropolitan areas.  Scott Turner, Level 3’s general manager for Central and Northern Florida said “Mid-market enterprises need a communications provider that knows the local marketplace and can provide the expert insight into how today’s network solutions – such as unified communications and collaborative technologies – can benefit their businesses.”  Level 3 reported fourth quarter and full year 2009 results last week with positive free cash flow of $44 million for the year. At the end of 2009 Level 3 had cash and cash equivalents of approximately $836 million. “In 2010, while we are still cautious about the effect the economy will have on our business, we believe it is the right time to increase our investment for growth,” said Sunit Patel, executive vice president and CFO of Level 3.

The plans are to offer service at a competitive price to at least 50,000 and potentially up to 500,000 people.  To assess interest they have setup a site with a request for information (RFI) to help identify interested communities.  Responses can be entered by local government entities or residents and community groups everywhere.

Google as your ISP?
Google doesn’t necessarily want to be your local Internet provider, but would like to shed some light on the true costs involved in bringing fiber to the home, and thus enabling 1gigabit per second connectivity to arrive faster. Stacey Higginbotham from GigaOm has additional details from Google’s product manager for alternative access Minnie Ingersol, who says Google will regularly push buttons in the industry to foster innovation and get things moving. It bid $4.6 billion in the FCC auction for U.S. wireless spectrum in 2008, and dabbled in the municipal WiFi arena by offering city-wide WiFi in its home base of Mountain View, California. The Google phone helped push the mobile industry along and separate the phone’s operating system from the carrier.  The  GigaOm interview ends with the question many asked – is Google going to become an ISP?  The answer -  this is a test bed for innovation and there are no plans to rollout a nationwide ISP network.

Rob Powell at Telecom Ramblings has an analysis of the innovation focus. “Google is trying to make a point about network architecture,” he wrties. “In fact it’s one that Dave Schaeffer of Cogent made to me recently.  There are huge differences in both functional capacity and economic viability between building a broadband access network over a legacy vs modern infrastructure.  It’s far easier to do (legacy) and that’s how most broadband is done in the US right now, but we really need to do (modern) to reach the kind of speeds everyone dreams of.” He also observes that “Google wants open access networks, and they’re willing to spend a large pile of cash to prove that they work.”

How large a pile? Barron’s Tech Trader Daily blog shares some cost estimates derived by Broadpoint.Amtech analyst Benjamin Schachter, who writes that “there are on average 2.6 people per household; so reaching 50,000 to 500,000 people means roughly 20,000 to 200,000 homes. The cost of the rollout – which will involve physically laying cable to individual homes – he thinks will be somewhere between $3,000 and $8,000 per household. That comes to anywhere from $60 million to $1.6 billion. At the midpoint – 100,000 homes, $5,000 each – you’d be talking $500 million.”

1 Gigabit per second!
Much of the discussion on the Internet is the uses for a 1 gigabit per second connection. The Google blog post announcing the news states things like downloading a high-definition, full-length feature film in less than five minutes or collaborating with classmates around the world while watching live 3D video of a university lecture.  Additionally they mention the possibility of medical imaging over the web, seeing what developers can do for next-generation applications and exploring new fiber network deployment techniques. Google Health was launched in order to “take charge of your health information.”  What if you could stream those three-dimensional medical images to your house?  Otherwise there are any number of rich-media, medical, gaming or business possibilities to consume the 1 gigabit pipe.

Broadband trends in the U.S.
ReadWriteWeb highlights broadband speeds around the world from a recent Akamai State of the Internet report. This is another key frustration for Google that they would like to address: In the third quarter of 2009 average broadband speeds in the U.S. declined by 2.4% compared to the same quarter in 2008.  Google has been active with the FCC’s National Broadband plan and pushing bold yet achievable goals.

Data is everything
As their ‘about us’ page explains, the name Google (googol) reflects the immense volume of information that exists, and the scope of Google’s mission: to organize the world’s information and make it universally accessible and useful.  There is a lot of data in the ISP realm and Google doesn’t want anything to get in the way of the end user’s Internet experience.  Prior to announcing the fiber network Google became a partner in the M-Lab open platform that allows researchers to deploy Internet measurement tools in order to enhance Internet transparency.  Built by researchers in Germany the M-Lab is set to gather data as well as uncover any discriminatory interference by a users’ ISP.  To assist in running the tests Google has volunteered 36 servers in 12 cities.

Google will be accepting responses to the fiber experiment until March 26, with the target communities announced later this year.

The $1 million award for the Greenscale Center for Energy-Efficient Computing at the University of California at Santa Barbara was the largest of 12 grants from the Google Focused Research Awards, which included support for machine learning, data collection via mobile devices and privacy, as well as computer energy efficiency.

The Greenscale award went to researchers Ricardo Bianchini of Rutgers, Fred Chong of UC Santa Barbara, Thomas Wenisch of the University of Michigan and Sudhanva Gurumurthi of the University of Virginia. The team will create a testbed to validate their concepts.

‘Radical Experiments’
“Greenscale will provide critical infrastructure to the project with the planned construction of the Greenscale Experimental Datacenter, a state-of-the-art miniature data center where systems researchers can conduct radical experiments not possible in production data centers,” said Fred Chong, professor of computer science at UCSB and the center’s director.

Bianchini has been studying computer energy efficiency since 2000, and is focusing on inefficiencies in the way servers use power. “Data centers have to be built to handle the highest anticipated demand,” he said. “But most of the time, they are only running between 20 and 50 percent of capacity. Trouble is, the computer servers in these centers consume about the same amount of energy whether their workload is low or high.”

The team will explore ways to create low power modes in servers, allowing parts of the computer to be turned off while other parts remain accessible. The goal is to allow less active servers to move their processing loads to other servers and essentially go to sleep. But information on the sleeping servers’ memories must still be instantly accessible.

Sleep Mode and Memory Access
In current computer designs, data requests go through a memory controller that is part of the central processing unit, or CPU. If that CPU is asleep, it cannot provide that access. The GreenScale team is seeking to redesign CPUs with a separate power feed to the memory controller, allowing it to perform the needed memory management functions while the rest of the CPU stays asleep. The goal of such redesigns would be to conserve 40 to 50 percent of the power that servers now consume.

The $1 million research grant from Google will fund two years of research, at which point Google will review the team’s progress and may award an additional $500,000 for a third year.

Other teams that received funding for research on energy efficiency in computing include:

• A Carnegie Mellon University team of computer science professors David Andersen and Mor Harchol-Balter will receive $100,000 to study scaling problems associated with building energy-efficient computing clusters.
• A University of Michigan team composed of Thomas Wenisch, Trevor Mudge, David Blaauw and Dennis Sylvester. Wenisch and Mudge are conducting research into the use of disaggregated memory to enhance the energy efficiency of data centers. Disaggregation refers to the separation of memory from other data center server components and linking together of those components through an interconnect.
• A Stanford University team including VMware co-founder Mendel Rosenblum, Christos Kozyrakis, Mark Horowitz, Benjamin Lee, and Nick McKeown.
• A project from Tajana Simunic Rosing, Steven Swanson and Amin Vahdat from the University of California at San Diego.
• Princeton University researchers Margaret Martonosi, Jennifer Rexford, Michael Freedman and Mung Chiang

First there’s this from The Register: “Sometime in the middle of October, Google silently launched a new net domain – a barely-disguised doppelgänger to the familiar google.com – and according to the latest stats from the site watchers at Alexa, this mystery domain is now visited by nearly three per cent of all net users, making it the 44th most visited domain on the interwebs.”

Separately, Network World notes that Google has “quietly turned on IPv6 support for its YouTube video streaming Web site, sending a spike of IPv6 traffic across the Internet that has continued from last Thursday until Monday. Industry observers say YouTube appeared to be supporting IPv6 in production mode, as opposed to running a test of the next-generation Internet protocol.”

The capex “breather” has been enabled by the company’s building boom in 2007-08, during which it announced major data center construction projects in Lenoir, North Carolina; Goose Creek, South Carolina; Pryor, Oklahoma and Council Bluffs, Iowa.

After the economic meltdown gripped Wall Street last fall, Google throttled back spending even further, opting to delay construction of its data center in Pryor.