Velocity: Flickr on Devs, Ops and Teamwork

There were many excellent presentations at the Velocity 2009 conference this week in San Jose, Calif. But the one that generated he most buzz was a talk by John Allspaw and Paul Hammond of the Flickr team at Yahoo titled “10+ Deploys A Day: Devs & Ops Cooperation at Flickr.” A summary: “Flickr takes the idea of ‘release early, release often’ to an extreme - on a normal day there are 10 full deployments of the site to our servers. This session discusses why this rate of change works so well, and the culture and technology needed to make it possible.” The Velocity team has posted full video of the session, which we’ve embedded below. The presentation is also online so you can follow along, but be warned that it’s a large PDF file.

For additional video, check out our DCK video archive and the Data Center Videos channel on YouTube.

Yahoo Unstealths Its Data Center Efficiency

The Yahoo data center in Quincy, Washington includes cooling-optimized "podules" with a PUE of 1.21 (Photo: Yahoo Inc.)

When it comes to data center efficiency, Yahoo has maintained a lower profile than rivals Google and Microsoft. But the Yahoo team is building a compelling data center story of its own, with innovations in cooling design and energy efficiency ratings approaching the best that Google has achieved.

Yahoo’s Adam Bechtel began telling the story yesterday at the O’Reilly Velocity 2009 conference in San Jose, Calif. Bechtel, the chief architect of Yahoo’s data center operations, shared details of a patented cold-aisle containment system that integrates an overhead cooling module, building the air conditioning units into the top of a “podule” of cabinets packed with servers.

That design has helped Yahoo lower its Power Usage Effectiveness (PUE) to 1.21, according to Bechtel, just a hair shy of the best numbers disclosed by Google and a slightly better than the lowest PUE reported by Microsoft. The PUE metric (PDF) compares a facility’s total power usage to the amount of power used by the IT equipment, revealing how much is lost in distribution and conversion.

Costs Driving Innovation
Bechtel notes that although PUE is a useful benchmark, Yahoo’s focus on efficiency was driven by the bottom line. “We were spending obscene amounts on infrastructure,” Bechtel recalled. “Our power consumption was doubling every 10 months, and that was just a shocker. We started to look at energy consumption in a very different way.”

Yahoo: Weather a Key Benefit of NY Location

In a sign of the growing importance of free cooling in data center site location, representatives of Yahoo cited the cool weather in Western New York as a factor in the company’s interest in building a $150 million data center project in Lockport. Yahoo senior director of operations Scott Noteboom said the company was planning for its “most efficient design to date,”  with an approach intended to lessen usage of water and power. Here’s a video report posted by Buffalo TV station WIVB.This video runs about 2 minutes, 15 seconds.

Yahoo Eyes Lockport, NY for Data Center

Yahoo (YHOO) has filed plans for a $150 million data center in Lockport, New York and hopes to begin work in August building a 190,000 square foot server farm. Officials in Lockport announced Yahoo’s interest after the company filed site plans with the city planning board. Lockport is about 25 miles northeast of Buffalo, NY.

Yahoo has told local media the Lockport location is “not a done deal” and the company is still considering other states. Yahoo’s board has yet to approve the proposal, and the company is apparently debating whether to buy or lease the site. But the filing of site plans suggests that a decision is all but done but can’t be finalized until a pending package of tax breaks is approved.

The company has filed site plans for a 30-acre parcel along Route 270. Yahoo also considered sites in nearby towns of Pembroke and Cambria, and had previously scouted locations in Virginia and eastern Canada.

NY Approves Power for Yahoo Data Center

New York State officials say a $150 million Yahoo data center is bound for the Buffalo area after the New York Power Authority approved a deal to supply the Internet company with 15 megawatts of power. But the company says it has made no commitments, and industry sources say other states are still in the hunt.

Gov. David Paterson said the decision must be approved by Yahoo’s board, but that he ”expects to secure Yahoo!’s commitment to invest in Western New York rather than in competing sites in other states.” The governor’s office said the new facility is expected to be located in either Niagara or Genesee Counties, and would create 125 jobs paying approximately $65,000.

But sources in the data center industry suggest Yahoo’s decision is not a done deal. Yahoo has scouted sites in at least two other locations. Although Virginia officials say the state’s recent approval of tax incentives for data centers was not designed to attract a specific deal, the measure offers tax exemptions for projects of at least $150 million - exactly the size of the Yahoo development.  

“We are continuously evaluating new ways to best support our business needs and we are in a dialogue with various state officials about our potential options,” Yahoo said in a statement to the Buffalo News. “New York State Officials and the Governor’s office have been very supportive in helping us evaluate our options in New York State, and we’re pleased to learn of the New York Power Authority’s vote.”  

Yahoo Considers NY State for Data Center

Yahoo is considering several sites in Western New York for a major data center, according to New York state officials, who are seeking to leverage the area’s supply of hydro power to attract the huge Internet company. Discussions between Yahoo and officials in Gov. David Paterson’s administration began about six weeks ago, according to the Buffalo News, which said the New York Power Authority (NYPA) has been directed to work with Yahoo on the company’s requirements.

“They are seriously, seriously considering a site in Orleans County or Genesee County,” Sen. Chuck Schumer told the News. “They are considering other sites … but Western New York is at the top of the list. If Yahoo locates in Western New York, it would be an eye-opener for many other types of firms to look at us.”

Genesee and Orleans counties are each east of Buffalo, not far from the Niagara River and its supply of hydro power, which has been a major attraction in helping central Washington state attract major data center projects - including one of Yahoo’s largest company-built facilities in Quincy, Wash.. But serious questions remain about the NYPA’s ability to deliver capacity to large industrial users. Last month Steel Development Co. abandoned plans to build a $200 million plant in Western New York when the NYPA could offer only 17.5 megawatts of capacity, half the 35 megawatts the company was seeking. Local media report that Google also considered a data center in the area, but was unable to finalize a deal with the NYPA.

Nebraska’s Wooing Led to Yahooing

Nebraska economic development officials have been working hard to attract major data center projects to their state. And it must have been frustrating to see Google and Yahoo each announce enormous data center projects next door in Iowa.

So when Yahoo came to Nebraska as part of a 17-state site location process, state officials were determined to compete hard for the project, a commitment that extended to the governor’s office. When company and local officials gathered this morning to announce Yahoo’s $100 million data center in La Vista and customer care center in Omaha, Gov. Dave Heineman wore purple Converse Chuck Taylor All-Stars. Purple, of course, is Yahoo’s signature color.

“The personal involvement of the governor was the key here, as evidenced by his purple shoes,” Yahoo Vice President of Operations Kevin Timmons told the Omaha World-Herald. “His involvement was a big factor compared to the competing states. I got a lot of calls from him asking, ‘Kevin, what do we have to do to get to a yes?’”

Yahoo Set to Unveil Nebraska Project

Yahoo has purchased this building in La Vista, Nebraska for $14.8 million.

Yahoo today will confirm its plans to build a major new data center facility in La Vista, Nebraska, a suburb of Omaha. Nebraska Gov. Dave Heineman is scheduled to hold a news conference at 10 a.m. in La Vista and is expected to discuss the state’s effort to win the project, which is expected to include an investment by Yahoo of at least $100 million. A key development was Nebraska’s adoption of tax incentives that made it more attractive for Yahoo to build in Nebraska than Iowa, which has already won huge data center projects for Google and Microsoft.

Yahoo recently paid $14.8 million to purchase a 300,000 square foot existing structure on 24.3 acres of land in La Vista. The building previously served as the headquarters for Tender Heart Treasures, which makes gifts and home decor products, and will vacate the building in January.

The new facility is a major step in Yahoo’s ongoing expansion of its data center infrastructure to support its online properties. The timing of the announcement is somewhat awkward, following this week’s announcement that Yahoo will lay off 1,400 workers as part of a plan to cut costs by $400 million. But Yahoo has been less aggressive in its data center capital spending than rivals Microsoft and Google, which have each announced more than $2 billion in new projects since the start of 2007.