Twitter [1] has begun usin file distribution technology from
BitTorrent to boost the performance of its back-end infrastructure, which has required constant attention to keep pace with the growth of the popular microblogging service, which has now reached
1 billion tweets per month [2], according to Pingdom.
“With BitTorrent, Twitter is planning to distribute files faster and more efficiently, saving time and precious resources and improving the scalability of Twitter’s operation,” writes Ernesto at
TorrentFreak [3] (via
Glyn Moody [4]).
Here’s the
full description [3]: “Twitter’s new project, codenamed ‘Murder’, will not use the bandwidth of Twitter users. Instead, it will transform the site’s servers into a large BitTorrent swarm that will distribute file updates using BitTorrent technology. The setup is pretty straightforward. Murder uses a ’seeder’ server where the new files will be distributed to thousands of ‘peer’ servers. Because all servers assist in the deployment of the files, it will only take a fraction of the time it would otherwise take when files are distributed from a central server.”
The TorrentFreak story described the project as being in the development stage. But Twitter staff have since confirmed that it is live, and the Murder code is
available on GitHub [5]. “Murder is already in production and has been for several months,” Twitter engineer Ryan King shared in our comments. “We use it to deploy all our code.”
Murder “allows for scaleable and fast deploys in environments of hundreds to tens of thousands of servers where centralized distribution systems wouldn’t otherwise function,” according to the GitHub description from Twitter’s Larry Gadea. It’s based on the BitTornado client, and requires one server to act as a tracker, and another to seed the files to the servers that will share them, distributing the load across tens of thousands of servers.
BitTorrent Inc., the company formed to commercialize the open source peer-to-peer software, expressed interest in helping other large Internet platforms explore the potential for BitTorrent to boost back-end file transfer.
Read more at
TorrentFreak [3]. For additional background reading, here’s some of our previous reporting on the the evolution of Twitter’s infrastructure:
Rich Miller is the founder and editor-in-chief of Data Center Knowledge, and has been reporting on the data center sector since 2000. He has tracked the growing impact of high-density computing on the power and cooling of data centers, and the resulting push for improved energy efficiency in these facilities.
Article printed from Data Center Knowledge: http://www.datacenterknowledge.com
URL to article: http://www.datacenterknowledge.com/archives/2010/02/10/twitter-using-bittorrent-to-speed-servers/
URLs in this post:
[1] Twitter: http://www.twitter.com
[2] 1 billion tweets per month: http://royal.pingdom.com/2010/02/10/twitter-now-more-than-1-billion-tweets-per-month/
[3] TorrentFreak: http://torrentfreak.com/twitter-uses-bittorrent-for-server-deployment-100210/
[4] Glyn Moody: http://twitter.com/glynmoody/status/8902199710
[5] available on GitHub: http://github.com/lg/murder
[6] Using Metrics to Vanquish the Fail Whale: http://www.datacenterknowledge.com/archives/2009/06/23/twitter-using-metrics-to-vanquish-the-fail-whale/
[7] Twitter’s Growth Drives NTT Expansion: http://www.datacenterknowledge.com/archives/2009/08/31/twitters-growth-drives-ntt-expansion/
[8] How Much of Twitter’s $100 Million is for Servers?: http://www.datacenterknowledge.com/archives/2009/09/28/how-much-of-twitters-100m-is-for-servers/
[9] Twitter in Combat Mode as Attacks Continue: http://www.datacenterknowledge.com/archives/2009/09/09/twitter-in-combat-mode-as-attacks-continue/
[10] Rich Miller: http://www.datacenterknowledge.com/archives/author/richm/
Click here to print.
