Digg Will Expand Its Data Centers

Digg will use some the $28 million in funding it announced last week to expand its data center infrastructure, according to company officials. The fast-growing social media site is optimizing its current infrastructure, but will soon need additional data center space.

“We currently have some room to grow but we’ll be expanding to a larger space in the coming months to fit the new growing infrastructure,” said Scott Baker, Digg’s VP of Operations. “We are always expanding our infrastructure to meet the demands of higher traffic and new features that we’re always developing.”

The company’s short-term goal is to refine its data center architecture to serve content from two locations, rather than just one. “We are currently in two data centers set up in an active/passive configuration for disaster recovery purposes,” said Baker. “Part of this expansion is to make better use of our (backup) data center installation. We are working towards some site rearchitecture to allow us to have an active/active setup with traffic being served from both locations simultaneously.”

How Digg Works: A Look Under The Hood

The social media site Digg is known for the huge volume of traffic it sends to sites that are linked on its front page. How does Digg manage its web site to support all that activity - 22 million users and 220 million page views in August, according to QuantCast)? Joe Stump provides an overview of How Digg Works in the first post on the Digg Technology Blog, which made its debut Thursday. An excerpt:

Ask Ron - our Systems Engineering Lead - the exact number of servers we have in production and he’ll probably respond with, “I don’t honestly know.” I can say we’ve got dozens of web servers and dozens more DB servers. I can say with certainty it takes six specialized graph database servers to run the Recommendation Engine and we have another six to ten machines that serve files from MogileFS. But really, the numbers are the least interesting part of the equation. What makes Digg an interesting place to work are what the pieces are and how they fit together.

Stump’s post shares some details on Dogg’s architecture and how the various pieces work together to process typical requests from Diggers like me.

Digg isn’t alone among social networking sites sharing information about their efforts on the back-end. The Engineering @ Facebook blog provides an inside look at some of the infrastructure challenges face by Facebook.

How Digg Optimizes for Huge Traffic

How does the social media site Digg manage more than 26 million visitors a month? Site administrator Ron Gorodetzky talks to System Management News about the architecture Digg uses on its web site, with a focus on database management techniques. An excerpt:

“The first pain point we hit was just database stuff. The first thing you’ll notice is when you start to grow these queries, the database can’t commit as much time to committing a certain query as it used to,” said Gorodetzky. “You’ll find the normal things that work, suddenly don’t. You’ll find that, one day, you’ll see a spike in your graphs telling you that something’s going slower. Once you do that, you get to the point where the database part is as fast as it can be, you cache things. You scale out your Web server so you have more resources there, generally caching and doing less work per request.”

Gorodetzky also talks about the challenges involved with image serving, especially the expanded use of thumbnails. The site runs on a LAMP stack (Linux-Apache-MySQL-PHP).

You can track what we’re up to at Digg here. If you’re interested in data center and cloud computing news, add us to your friend list. Also, if you haven’t yet seen System Management News, it’s definitely a read. One of its columnists is John Rath, who many of you may know from his Data Center Links blog.

Are You A Digg User? DCK on Digg

In our never-ending quest for data center news, I’ve checked out most of the hot social media sites. Since Data Center Knowledge keeps me busy, it’s hard to find the time to be meaningfully active in more than one or two at a time. One I’ve stuck with is Digg, which was co-founded by a data center veteran (Jay Adelson of Equinix). If you’re a Digg user and Data Center Knowledge reader, check out our Digg profile and add us to your friends list. I’m interested in seeing what our DCK readers are Digging.

Digg Preps Disaster Recovery Site

Digg is preparing a disaster recovery data center that will allow the company’s IT team to quickly restore the site in the event that a disaster or network event knocks the primary data center offline. The announcement followed several hours of downtime after a planned site maintenance outage took longer than expected.

“We’ve been working for several months on a full Disaster Recovery site for Digg,” co-founder Kevin Rose wrote on the Digg blog. “Once operational, our DR site will let us make site updates without interruptions - and in tonight’s case where things went haywire, you’d never know we were making updates at all.”

That’s an important goal for Digg users, some of whom tracked the downtime and some of whom expressed their distress - and naturally blogged about it and submitted the URL to Digg as soon as it came back online.

Adelson Speaks About Equinix, Digg

Computerworld has an interview with Jay Adelson, the CEO of Digg.com, one of the Web’s most popular social media sites. Adelson was also a founder of Equinix, which operates data centers for network peering, and was asked about the parallels between the two companies. Adelson says the common thread is the disintermediation of incumbent technologies and companies:

(Digg) is similar to how Equinix (eliminated intermediaries) for telecom companies. When the Internet became a commercial medium in 1994, all of the Internet had been funded and operated by the government and universities. When it switched over to one operated by telecommunication companies, a very strong hierarchy developed. Tier 1 ISPs, the top five players in the world, would collect a dime on every packet that flowed throughout the Internet. Part of the reason was all the Internet networks had to interconnect with each other using these antiquated network access points operated by carriers. Equinix replaced these single, network-owned facilities with Internet business exchanges where anyone could exchange packets with anyone in a neutral playing field. This allowed the dot-coms like Yahoo and Google and others to really exert their might. Digg does a very similar thing to the media.

For more, including Adelson’s thoughts about the recent “user rebellion” at Digg, read the entire interview.

Equinix Provides Sturdy Back End for Digg

Recent enhancements at Digg have generated waves of additional web traffic for the popular social content hub, to the point where some measurement sites place it ahead of the New York Times and Slashdot. That kind of popularity can wreak havoc on the back end, as many fast-growing sites find their web infrastructure straining to keep pace.

Not so with Digg, which houses its servers at Equinix, a leading provider of network-neutral data centers and Internet exchange services. As traffic has scaled up, Digg has benefitted from Equinix’ peering infrastructure, which offers “immediate access to every major global network for the most efficient delivery of content to end-users.” Digg is housed in one of the company’s Silicon Valley data centers (Equinix doesn’t say which, remember the Fight Club rule).

Digg’s choice of Equinix is no shocker, as the companies have a shared lineage. Jay Adelson was the founder and chief technology officer at Equinix as it built its network and reputation. Adelson then joined with Kevin Rose to co-found Digg, which has quickly become one of the web’s most popular destinations. Adelson’s knowledge of Internet peering and traffic management - he was also a key player in the Palo Alto Internet Exchange (PAIX) before founding Equinix - made it a pretty safe bet that Digg would scale well if things took off. And they have, as Digg currently serves more than 8 million page views a day.