Twitter Using BitTorrent to Speed Servers

Twitter has begun usin file distribution technology from BitTorrent to boost the performance of its back-end infrastructure, which has required constant attention to keep pace with the growth of the popular microblogging service, which has now reached 1 billion tweets per month, according to Pingdom.

“With BitTorrent, Twitter is planning to distribute files faster and more efficiently, saving time and precious resources and improving the scalability of Twitter’s operation,” writes Ernesto at TorrentFreak (via Glyn Moody).

Here’s the full description: “Twitter’s new project, codenamed ‘Murder’, will not use the bandwidth of Twitter users. Instead, it will transform the site’s servers into a large BitTorrent swarm that will distribute file updates using BitTorrent technology. The setup is pretty straightforward. Murder uses a ’seeder’ server where the new files will be distributed to thousands of ‘peer’ servers. Because all servers assist in the deployment of the files, it will only take a fraction of the time it would otherwise take when files are distributed from a central server.”

Roundup: Force 10 Networks, Avocent, Facebook

Here’s a roundup of news announcements from the data center and hosting industry:

• Force 10 demonstrates at Interop. Force 10 Networks partnered with AFORE Solutions and Sun Microsystems at INTEROP in New York last week to demonstrate data center virutalization products and cloud computing initiatives focused on driving greater network agility. The 3 vendors demonstrated virtualized resource allocation for cloud-oriented applications as well as the AFORE ASE3300 Virutalization Extension Platform.  The ASE3300 and Force 10 switch and router solutions combine to enable a multi-site, virtual data center enabling migration to cloud computing environments.
• Avocent upgrades data center management software. Announced at Interop last week, Avocent is upgrading its MergePoint Infrastructure Explorer to include several new management capabilities.  The company said these enhancements will provide a unique view into capacity planning, bringing additional return on investment and total cost of ownership benefits.  Avocent CTO Ben Grimes said that the software will allow “customers to know where their assets are, as well as plan for different ‘what if’ scenarios, and manage their data centers to reduce risk -  all while bringing improved ROI and total cost of ownership benefits to customers.”  New features include rack timeline and an enhanced change management and capacity search capabilities.

Twitter is Frozen, and Then Thaws

What if the real-time Internet gets stuck? That seems to be what happened for several hours today with Twitter, as a large number of users reported that their timelines were not updating. The problems appeared to clear up around 3 pm Eastern time, when updates returned to a more normal pattern.       

Twitter acknowledged the unusual problems. ”Timelines remain stale for users,” the service reported on its status page. “We are deploying fixes to address the problem.” During the episode, users were able to tweet, and some of those messages appeared on Twitter search and third-party services like FriendFeed, according to TechCrunch.

Who cares if a few million Tweets get stuck in the space/time continuum? As we’ve previously noted, Twitter has become an important secondary support channel for many hosting and data center providers. As such, the service’s continued reliability problems have business implications.

How Much of Twitter’s $100M is for Servers?

One of the hottest tech stories last week was the $100 million raised by Twitter in a new round of funding, which reportedly values the microblogging service at $1 billion, even though it has yet to generate meaningful revenue. Much of the commentary has focused  on the high valuation, but the funding begs another question: Did Twitter raise $100 million because it can, or because it really needs the money?

Some analysts foresee Twitter using the money to hire 200 to 500 developers, or to acquire some of the companies building apps and services atop the Twitter API. But the real answer to the “need vs. want” question may depend on how much Twitter is spending on the infrastructure to support its rapid growth.

Massive Scalability Never Cheap
By outsourcing its infrastructure to NTT America and Amazon Web Services, Twitter has undoubtedly spent far less on infrastructure than if it built its own data centers. But massive scability never comes cheap, and as Twitter has gone mainstream, the rapid growth in its user base has meant additional investment in servers to handle the load.

Twitter’s storage requirements aren’t as onerous as some other social media sites, such as  Facebook, which stores billions of photos and videos. Twitter’s storage output is primarily text (Tweets), avatars and background images. Twitter saves money on image storage by using Amazon S3, but as its user base moves into the tens of millions, those costs will start to add up quickly.

Twitter in ‘Combat Mode’ As Attacks Continue

One of the famous Twitter error graphics (pre "Fail Whale"). Twitter has been enhancing its defenses against attacks.

Twitter has been the target of additional denial of service attacks in the wake of a major outage on Aug. 6, but enhanced defenses implemented by Twitter and NTT America have reduced their impact.

“We’ve been continually seeing attacks, but I think the public has not noticed because we’ve been mitigating them,” said Kazuhiro Gomi, Chief Operating Officer of NTT America, which hosts Twitter’s infrastructure. “We are definitely in combat mode, but handling it.”

Twitter was badly hobbled by the Aug. 6 attack, which also targeted Facebook and several other social media sites. In a recent interview, NTT America addressed the steps taken to harden Twitter’s defenses against these attacks, and addressed some criticisms of Twitter’s performance.

The Aug. 6 attack was a distributed denial of service (DDoS), in which “bot networks” of thousands of compromised computers are used to attack a site and overload its infrastructure – essentially clogging the pipes with too many requests at once. Network tools can be used to filter incoming traffic and deflect suspicious packets while forwarding “good” requests.

Defense Tools Turned On
These DDoS defense tools were available from NTT America, but Twitter did not have them in place at the time of the Aug. 6 attack. Once they were turned on, Twitter gradually improved its perfromance and returned to service. NTT America has since put some new defense mechanisms in place to address the unusual scope of recent attacks. These measures are not a customized solution for Twitter, but a broader hardening to protect all of NTT’s customers, the company said.

Twitter’s Growth Drives NTT Expansion

Driven by the rapid growth of Twitter, NTT America is expanding its data center footprint. NTT America confirmed today that it has leased 15,000 square feet of data center space in Santa Clara, citiing “rapidly growing customer demand” for data center resources.

That capacity crunch has been driven by Twitter, which outsources its infrastructure to NTT America through a managed hosting agreement. Twitter experienced 752 percent traffic growth in 2008, but that was nothing compared to the traffic explosion in the first half of 2009, as the microblogging service went mainstream and was widely adopted by celebrities and a growing number of businesses.

NTT America Chief Operating Office Kazuhiro Gomi said in June that “traffic generated by Twitter is getting so big, it’s basically eating up a lot of our data center network resources, especially the segment where Twitter is hosted. Other customers are riding on the same segment” in the company’s primary data center in San Jose, he said.

Twitter Postpones ‘Critical’ Maintenance

Sometimes the best laid plans of network engineers are overtaken by an international crisis. The microblogging service Twitter said late today that it would postpone scheduled maintenance downtime to keep the service available during the tumultuous political events in Iran. The move followed a day-long protest by some Twitter users, who said Twitter had become a critical communications tool for the Iranian opposition.

“A critical network upgrade must be performed to ensure continued operation of Twitter,” co-founder Biz Stone wrote on the Twitter blog. “In coordination with Twitter, our network host had planned this upgrade for tonight. However, our network partners at NTT America recognize the role Twitter is currently playing as an important communication tool in Iran. Tonight’s planned maintenance has been rescheduled to tomorrow between 2-3p PST (1:30a in Iran).”

This is yet another reminder of the critical importance of uptime, and how the universe of sites and services that “can’t go down” continues to expand. It’s a trend that places further focus on the data centers supporting these sites.

Twitter’s Servers Say ‘Hi!’

Is Twitter being powered by a stealth web server? Or is its tech team just friendly? The question is raised by Netcraft, which conducts a monthly survey of the Internet to track the number of web sites and the technologies they use. Netcraft’s Web Server Survey has been compiling growth data since August 1996, using an agent similar to a search “bot” that scoots around the web, gathering information on more than 230 million web sites. When it encounters a site, the Netcraft agent asks “What server software are you running?” Most times, the server will answer “Apache” or “Microsoft IIS.”

But not Twitter, which just says “hi.” This practice, known as server spoofing, is seen in Netcraft’s Twitter.com site info. Twitter is not the only site to intentionally alter server headers, with security concerns cited as a motivation in some instances. In other cases, services have used spoofing in a bid to influence Netcraft’s server market share data. Others are more playful, such as the group of sites identifying their server as ZX_Spectrum/1997 (Sinclair_BASIC), which was an 8-bit personal home computer released in 1982, but it was discontinued two years before the World Wide Web even existed.

For what it’s worth, the hosting history in Netcraft’s site report suggests Twitter is running a particularly sociable version of Apache.

The Downtime Is Back at Twitter

The improved performance at Twitter in recent months has led some observers (including DCK) to applaud the microblogging service’s new-found stability. It seems we spoke too soon. Twitter was down yesterday, and   unavailable again for several hours today (as documented by Pingdom).

The service, which became renowned for both its addictive nature and frequent downtime, has seen its traffic surge amid interest in the U.S. election, leading TechCrunch to observe that Twitter “is approaching escape velocity. If it doesn’t break up from all the pressure and is able to keep its service up and running more or less, it could soon – gasp! – break into the mainstream.”   

The Twitter team has given its users something to blog about while they’re gone in the form of a bizarre graphic that has replaced the oft-mocked “Fail Whale.” One more to add to the gallery of Twitter 404 art.