When Servers Lag, Virtual Assets Crash Too

Virtual goods are a growing sector of the digital economy, playing a large role in the developing business models for social networks like Facebook. Internet users in Asia spend approximately $5 billion a year on virtual goods created and traded in online games and apps, and the U.S. economy is expected to reach a similar size within five years.

So what happens when all these virtual assets are put in play and the servers can’t handle the load? A post at Slashdot provides a cautionary tale about the importance of sturdy infrastructure to support virtual economies. Players in the science fiction MMOG EVE Online claim that thousands of dollars worth of in-game assets were destroyed when server lag created an uneven playing field in a huge online battle between warring fleets of starships. Game operator CCP Games recently launched an expansion pack, and some gamers say the new code has experienced performance problems (CCP is running stress tests today to investigate the claims). Scandal or sour grapes? As one player noted on one of the EVE forums, “You don’t hear the winning side whining about lag very often.”

Interestingly, the debate about EVE Online emerges the same day that another space-themed MMO, Planet Calypso, announced that a player had paid $330,000 to buy a virtual space station.

WoW’s Back End: 10 Data Centers, 75,000 Cores

It takes a lot of resources to host the world’s largest online games. One of the largest players in this niche is Blizzard, which operates World of Warcraft and the Battle.net gaming service for its Starcraft and Diablo titles. World of Warcraft (WoW) is played by more than 11.5 million users across three continents, requiring both scale and geographic scope.

Blizzard hosts its gaming infrastructure with AT&T, which provides data center space, network monitoring and management. AT&T, which has been supporting Blizzard for nine years, doesn’t provide a lot of details on Blizzard’s infrastructure. But Blizzard’s Allen Brack and Frank Pearce provided some details at the recent Game Developer’s Conference in Austin. Here are some data points:

• Blizzard Online Network Services run in 10 data centers around the world, including facilities in Washington, California, Texas, Massachusetts, France, Germany, Sweden, South Korea, China, and Taiwan.
• Blizzard uses 20,000 systems and 1.3 petabytes of storage to power its gaming operations.
• WoW’s infrastructure includes 13,250 server blades, 75,000 CPU cores, and 112.5 terabytes of blade RAM.
• The Blizzard network is managed by a staff of 68 people.
• The company’s gaming infrastructure is monitored from a global network operating center (GNOC), which like many NOCs, features televisions tuned to the weather stations to track potential uptime threats across its data center footprint.

The AT&T Gaming Core Team was formed in 2004 to host gaming operations using AT&T’s IP network. The team consists of engineers and hosting specialists who provide round-the-clock support to companies offering MMO games.

Virtual Goods and the Cost of Infrastructure

Some of the customer racks supporting online gaming operations inside a GNi data center.

In an economy built atop virtual goods, how do you manage capacity and sort out whether the business model is able to pay for the infrastructure required to power it? That’s an important questionas as virtual item purchases become a key revenue stream for fast-growing social networking platforms like Facebook.

Managed hosting specialist GNi, which offers an Infrastructure as a Service (IaaS) hosting platform for online games, offers a window into the evolving infrastructure requirements of virtual economies. In the U.S., popular understanding of online gaming has been shaped by the popularity of World of Warcraft and other virtual worlds that generate through monthly subscription fees. That approach is increasingly yielding to a “free-to-play” business model.

“A lot of games are free, and now make their money by selling virtual items,” said Robert Yusin, General Manager of Field Operations at GNi. “The most popular games that we see are the virtual item games.”

Pioneered in Asia
This gaming business models based on virtual goods developed in Asia, where some industry watchers say virtual goods is a $7 billion market. One of GNI’s large gaming customers is Frogster Interactive, whose free-to-play game Runes of Magic has more than 2 million users.

Will COD Switch Slow Game Server Demand?

Online gaming has always been an important niche in the dedicated server market, generating business for game server providers (GSPs) as well as larger “mass market” dedicated hosts. One of the leading titles in this arena has been the “Call of Duty” series from Infinity Ward, which revealed this week that its upcoming Call of Duty: Modern Warfare 2 will not support dedicated servers, but will instead run on a dedicated online platform operating by the company.

“In effect, gamers in the PC community will not be able to run Modern Warfare 2 on dedicated servers that they may rent, as they are able to with previous Call of Duty games,” writes GamaSutra. “In addition, no dedicated servers likely means that IWNet will not support mods and custom maps for the game, at least at launch.”

Second Life Adds East Coast Servers

The NAP of the Capital Region in Culpeper, Virginia.

Linden Lab will expand the back-end for its Second Life virtual world to the East Coast, adding server infrastructure at the NAP of the Capital Region, a large data center operated by Terremark Worldwide in Culpeper, Virginia. The colocation agreement will help Second Life extend its 3D virtual world environment to support additional users.

Second Life was launched in 2003, and has become a large hub for user-generated content, with an “in-world” economy with more than $450 million in annual user-to-user transactions. The primary hosting facilities supporting the game are in San Francisco and Oakland. The future growth of Second Life was a factor in Linden’s choice of Terremark and the NAP of the Capital Region, a technology campus designed to eventually house five large data centers.

Second Life Scouts UK Data Centers

The operators of the Second Life virtual world are scouting locations for data center space in the United Kingdom, according to Massively, which tracks the MMO gaming sector. Linden Labs is said to be shopping for high-density space to house an expansion of its infrastructure for Second Life, which currently houses most of its servers in a pair of data centers in the San Francisco Bay Area. 

Massively notes that a geographic expansion of Second Life might bring both benefits and challenges, given the virtual world’s architecture and historic scalability issues. ”It would be the first outside of the USA, and provide significantly faster access for EU residents to assets and simulators (aka regions) that were hosted at the facility,” the story notes. “By the same token, grid stability tends to be compromised when any pair of the Lab’s data-centers are unable to communicate with each-other, and an additional facility would seem to double the potential risk of such troubles.”

When In Rome

Hosting gaming infrastructure is becoming a big business. That was clear at the Game Developers Conference at the Moscone Center in San Francisco, where Terremark Worldwide (TMRK) was on hand with a booth customized for the gaming crowd that showcased its NAP of the Americas in Miami. Photo by Cory Doctorow.

World of Warcraft Will Stay on AT&T Servers

Some customer renewals are more important than others. Hosting deals don’t get much bigger than AT&T’s contract to host the infrastructure for World of Warcraft, the world’s largest massively multiplayer online game (MMOG) with more than 11.5 million players worldwide. AT&T said Tuesday that game publisher Blizzard Entertainment has agreed to a two-year renewal to host World of Warcraft in North America.

AT&T has been working closely with Blizzard Entertainment for nine years, providing hosting for World of Warcraft and the Battle.net online-gaming service. “We’ve been pleased with AT&T’s service over the years, and we’re confident in their ability to continue to provide the quality support we need,” said Paul Sams, Chief Operating Officer of Blizzard Entertainment. “This renewed agreement allows us to stay completely focused on our games while AT&T applies its hosting expertise to help us offer the speed, reliability and security our players expect.”

GameRail Network Shuts Down

GameRail, the private gaming backbone providing low-latency connections to gamers, has announced that it is shutting down:

It is with deep regret to announce that the GameRail network has been discontinued at this time. … We believe that latency and network quality will continue to affect the gaming experience and while we are still believers in the GameRail concept, the market does not appear to be ready to support a standalone network for gaming at this time.

GameRail was a customer of Internap (INAP), which provided the gaming network with intelligent routing services to support its service. It also had connectivity agreements with XO Communications and Broadwing. The network was based in the Bandwidth Exchange buildings in St. Louis, which are now owned by Digital Realty Trust (DLR).