Managing Megasites: ‘An Insane Amount of Will’

One runs a popular service on just 350 servers, while another likely has more than a million servers. The common denominator: major traffic. Executives from six of the web’s most popular properties - Google, Microsoft, Yahoo, Facebook, MySpace and LinkedIn - shared the stage at Structure 09 yesterday to discuss their infrastructure and innovations.

Managing a megasite requires plenty of hardware. But that’s not the secret sauce, according to Vijay Gill, the Senior Manager of Engineering and Architecture at Google (GOOG). “The key is not the data centers,” said Gill. “Those are just atoms. Any idiot can build atoms and have this vast infrastructure. How you optimize it - those are the hard parts. It takes an insane amount of will.”

The challenges faced by the six sites varied. “I’m taking a minimalist approach,” said Lloyd Taylor, the VP of Technical Operations for the LinkedIn social network. “How little infrastructure can we use to run this? The whole (LinkedIn) site runs on about 350 servers.” That’s due largely to the fact that much of content served by LinkedIn consists of profiles and discussion groups are heavy on text. “We’re not a media intensive site,” said Taylor.

The Billion Dollar HTML Tag

Marissa Mayer of Google discusses web site performance Wednesday at the Velocity 2009 conference (photo by James Duncan Davidson via Flickr).

Can a single HTML tag really make a difference on a corporation’s financial results? It can at Google, according to Marissa Mayer, who says web page loading speed translates directly to the bottom line.

“It’s clear that latency really does matter to users,” said Mayer, the VP of Search and User Experience at Google and today’s keynote speaker at the O’Reilly Velocity Conference. Google found that delays of fractions of a second consistently caused users to search less. As a result, its engineers consistently refine page code to capture split-second improvements in load time.

This phenomenon is best illustrated by a single design tweak to the Google search results page in 2000 that Mayer calls “The Billion Dollar HTML Tag.” Google founders Sergey Brin and Larry Page asked Mayer to assess the impact of adding a column of text ads in the right-hand column of the results page. Could this design, which at the time required an HTML table, be implemented without the slower page load time often associated with tables?

Mayer consulted the W3C HTML specs and found a tag (the “align=right” table attribute) that would allow the right-hand table to load before the search results, adding a revenue stream that has been critical to Google’s financial success.

Voldemort Industries

Google is serious about its data centers. But it’s also determined not to take itself too seriously, as evidenced by an anecdote in Oregon Business News, which examines the impact of economy in a Google data center on the economy in The Dalles, Oregon. “As you pull up to the riverfront campus, you’ll spot a Voldemort Industries sign, a self-effacing reference to the Harry Potter character known as ‘He-Who-Must-Not-Be-Named,’” writes The Oregonian.

Okay, perhaps it’s not perfectly aligned with the whole “don’t be evil” motto. The sign is a reference to the use of a code name (Project 02) by local officials during the planning process for the Google data center in The Dalles. The whole cloak-and-dagger business apparently didn’t sit well with some in the community. 

“Google officials say they learned from the backlash, and make a point to be transparent when they open data centers,” the story notes. “They have also gotten involved in The Dalles. Workers volunteer at cleanups or Habitat for Humanity; a garden at the edge of the property is public; grants go to community groups. Last fall, Google hosted an open house at its cafeteria and visitor center.”

The good news - while the presence of Google hasn’t transformed The Dalles into a data center hub, it appears to have helped blunt the impact of the economic downturn in The Dalles, which has also benefited from the growth of local wind farms. Here’s a video from The Oregonian with several vignettes about the economy in The Dalles, including a look inside the office and cafeteria of the Google data center:

Vint Cerf at the Google Internet Summit

In early May Google hosted the Google Internet Summit 2009 at its Mountain View, Calif. campus. The event brought together thought leaders in Internet infrastructure, with the goal of gathering “a wide range of knowledge to inform Google’s future plans.” This video presents the introductory remarks from Vint Cerf, Google’s chief Internet evangelist and the co-designer of the TCP/IP network protocol. ”The Internet is at an important flex  point in its history,” Cerf says. ”Scaling, in many different ways, is still an important issue. Cloud computing is adding another dimension to the way the Internet is being used.” This video runs 9 minutes, but the substance of Cerf’s remarks commece at the 4 minute mark after some opening greetings and housekeeping. 

For more news from Google, visit our Google Channel. For additional video, check out our DCK video archive and the Data Center Videos channel on YouTube.

Google Opens Council Bluffs Data Center

About 650 people attended a ceremony yesterday to mark the “launch” of Google’s new data center in Council Bluffs, Iowa. The $600 million facility is seen as a key economic development win for Iowa, and will eventually result in 200 full-time jobs paying $50,000 a year. “Google has opened up the door to opportunities for us - and Iowa - that we didn’t have in the past,” Council Bluffs Mayor Tom Hanafan said Tuesday, when state and local leaders, workers and about 100 local residents gathered at the new facility. “All of a sudden, companies are looking at Iowa a little differently. Google has put us on the map.”

Google is still hiring workers in Council Bluffs. but local officials noted that the company has room to expand. Google has purchased an additional 1,000 acres of land about four miles from the first phase of its data center project. The company was considering an adjacent 130-acre piece of land for the second phase, but may eventually expand at the larger parcel instead. The purchase doesn’t necessarily mean that Google is expanding the scope of its project, but gives it the space to build additional data centers if needed.

See the Omaha World-Herald and Des Moines Register for additional coverage and photos.

Google on ‘The Data Center as a Computer’

A photo of the Google data center in Goose Creek South Carolina.

We’ve written often about Google’s data center operations, which have include innovations such as data center containers and custom servers featuring on-board UPS batteries. Google’s approach to design and innovation are shaped by a vision of the data center that extends beyond traditional thinking. Two of the company’s data center thought leaders, Luiz Andre Barroso and Urs Holzle, have published The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines (PDF), a paper that summarizes the company’s big-picture approach to data center infrastructure.

“As computation continues to move into the cloud, the computing platform of interest no longer resembles a pizza box or a refrigerator, but a warehouse full of computers,” write Barroso and Holzle. “These new large datacenters are quite different from traditional hosting facilities of earlier times and cannot be viewed simply
as a collection of co-located servers.

Google Traffic Shifted to NTT During Outage

What happened during Thursday’s performance problems for Google? The company said that “an error in one of our systems caused us to direct some of our web traffic through Asia, which created a traffic jam.” Renesys, which tracks Internet routing, has additional details in a blog post today.

Renesys says traffic was shifted to NTT, whose network received an influx of traffic bound for Google that would normally be routed through Level 3 and/or AT&T. At one point NTT’s network was handling 85 to 90 percent of the traffic bound for Google, according to Renesys. Check out Martin Brown’s analysis for more.

NTT America said that traffic flow may have shifted, but the problems were due entirely to issues at Google.  ”NTT’s network was not the cause of Google’s performance problems,” said a spokesperson for NTT America, which maintains the company’s network. “No traffic jam occurred at NTT.”

Rolling Outage for Google

Many users are experiencing trouble reaching Google today in a rolling outage that is affecting some regions more than others. The troubles were first seen at Google News, which came back online after an outage this morning, apparently to add video links to news searches.

Meanwhile, there are widespread reports on Twitter of trouble reaching other Google services, and even including the home page. The Google Apps status page is acknowledging a “service disruption” for Gmail and says a problem with Google Calendar has been resolved.

UPDATE: Urs Holzle, who oversees the company’s data center operations, has posted an explanation on the official Google blog. ”An error in one of our systems caused us to direct some of our web traffic through Asia, which created a traffic jam,” Holzle wrote. “As a result, about 14% of our users experienced slow services or even interruptions. We’ve been working hard to make our services ultrafast and ‘always on,’ so it’s especially embarrassing when a glitch like this one happens. We’re very sorry that it happened, and you can be sure that we’ll be working even harder to make sure that a similar problem won’t happen again.”