Posted By Rich Miller On February 10, 2010 @ 9:21 am In Twitter | 3 Comments
Twitter  has begun usin file distribution technology from BitTorrent to boost the performance of its back-end infrastructure, which has required constant attention to keep pace with the growth of the popular microblogging service, which has now reached 1 billion tweets per month , according to Pingdom.
“With BitTorrent, Twitter is planning to distribute files faster and more efficiently, saving time and precious resources and improving the scalability of Twitter’s operation,” writes Ernesto at TorrentFreak  (via Glyn Moody ).
Here’s the full description : “Twitter’s new project, codenamed ‘Murder’, will not use the bandwidth of Twitter users. Instead, it will transform the site’s servers into a large BitTorrent swarm that will distribute file updates using BitTorrent technology. The setup is pretty straightforward. Murder uses a ’seeder’ server where the new files will be distributed to thousands of ‘peer’ servers. Because all servers assist in the deployment of the files, it will only take a fraction of the time it would otherwise take when files are distributed from a central server.”
The TorrentFreak story described the project as being in the development stage. But Twitter staff have since confirmed that it is live, and the Murder code is available on GitHub . “Murder is already in production and has been for several months,” Twitter engineer Ryan King shared in our comments. “We use it to deploy all our code.”
Murder “allows for scaleable and fast deploys in environments of hundreds to tens of thousands of servers where centralized distribution systems wouldn’t otherwise function,” according to the GitHub description from Twitter’s Larry Gadea. It’s based on the BitTornado client, and requires one server to act as a tracker, and another to seed the files to the servers that will share them, distributing the load across tens of thousands of servers.
BitTorrent Inc., the company formed to commercialize the open source peer-to-peer software, expressed interest in helping other large Internet platforms explore the potential for BitTorrent to boost back-end file transfer.
Read more at TorrentFreak . For additional background reading, here’s some of our previous reporting on the the evolution of Twitter’s infrastructure:
Article printed from Data Center Knowledge: http://www.datacenterknowledge.com
URL to article: http://www.datacenterknowledge.com/archives/2010/02/10/twitter-using-bittorrent-to-speed-servers/
URLs in this post:
 Twitter: http://www.twitter.com
 1 billion tweets per month: http://royal.pingdom.com/2010/02/10/twitter-now-more-than-1-billion-tweets-per-month/
 TorrentFreak: http://torrentfreak.com/twitter-uses-bittorrent-for-server-deployment-100210/
 Glyn Moody: http://twitter.com/glynmoody/status/8902199710
 available on GitHub: http://github.com/lg/murder
 Using Metrics to Vanquish the Fail Whale: http://www.datacenterknowledge.com/archives/2009/06/23/twitter-using-metrics-to-vanquish-the-fail-whale/
 Twitter’s Growth Drives NTT Expansion: http://www.datacenterknowledge.com/archives/2009/08/31/twitters-growth-drives-ntt-expansion/
 How Much of Twitter’s $100 Million is for Servers?: http://www.datacenterknowledge.com/archives/2009/09/28/how-much-of-twitters-100m-is-for-servers/
 Twitter in Combat Mode as Attacks Continue: http://www.datacenterknowledge.com/archives/2009/09/09/twitter-in-combat-mode-as-attacks-continue/
 Rich Miller: http://www.datacenterknowledge.com/archives/author/richm/
Copyright © 2012 Data Center Knowledge. All rights reserved.