Building Google's 'Insane' Infrastructure
December 21st, 2007 By: Rich Miller
Dan Farber at ZDNet has an interesting video interview with Lloyd Taylor, the Vice President of Technical Operations at LinkedIn. Taylor joined LinkedIn after serving as director of global operations for Google (GOOG), where he helped the search giant develop its data center infrastructure. Taylor talks about the challenges Google faced as its business began to scale:
The problem at Google, when I joined, was that it was no longer possible to get enough colocation space to handle the growth of the company. The size of the computing infrastructure that Google uses is insanely large. A surprisingly small percentage of it is responsible for delivering end user services. Generally, you want to put that close to the end users. So, the idea of having smaller data centers around the world to provide local access is one part of the strategy.
But the need for speed in delivering search results posed some unique challenges in approaching that distributed data center infrastructure. Taylor says Google needed to “insanely massive amounts of computing so that you can run these very complex and very highly parallel algorithms on the web data. So, a lot of the challenge was how do we get enough space and how do we do it in an affordable way.”
The operations team wound up taking a new stripped-down approach to virtually every element of their data center design and engineering, according to Taylor:
What we did at Google is essentially throw out the book on how to build data centers, and went back to basic heat transfer theory, went back to basic electrical theory, and essentially threw away everything that wasn’t strictly necessary in sort of a minimalist design. Essentially the integrated circuit industry concept is step and repeat.
So, instead of building chips on a silicon surface, what we figured out how to do was to buy 1,000 acres in Oklahoma, and then build the data center. And when we need more space, actually be able to step and repeat that data center, build another one right next to it. And so that is the environment they have now. So, they have large amounts of land in various places around the world. And as the business needs more space, they have a mechanism where they can simply start from scratch and have a new data center up and running in less than six months.
That last statement caused Farber to double-check. “In less than six months?” he asked. Yes, Taylor confirmed, less than six months.
Taylor also talks about LinkedIn. You can watch the entire video (it runs about 12 minutes) on ZDNet.
Frank A. ColuccioPosted December 22nd, 2007
I sometimes muse over the prospects of current-day data center builders overshooting their eventual needs. During the mid-Eighties, for example, I came into contact with many situations where multiple-soccer-field-sized buildings were being constructed that now serve as produce centers and parking lots, owing to the ongoing effects of miniaturization on older generations of computer mainframes and big iron telecoms infrastructure. Even today, one is more apt to run out of power — if not a limitation of in-building power capacity, then from the limitations of the regional grid — before they run out of space in many data centers. Tere’s still a ways to go, granted, before computing and networking elements are reduced to the size and form factor of the interface, which is where I tentatively believe technology is headed. Another factor to consider is the proximity of data centers to the end users they serve, which is an invincible and all-important factor in reducing propagation times in the fulfillment of queries and transactions, which goes against the grain of maintaining only a few regional centers feeding the universe over greater distances. All of these factors combine to beg the question of whether some of today’s humongous data centers themselves may not one day become the brick-and-mortar components of outfits such as Amazon and Fresh Direct. Thoughts?