Using Metrics to Vanquish the Fail Whale

John Adams of the Twitter ops team discusses the use of metrics to improve web site performance at Velocity 2009 (Photo by James Duncan Davidson via Flickr)

Few prominent web sites have failed more often and under closer scrutiny than Twitter. But over the past year the microblogging service has rehabilitated its reputation, improving its uptime even as its traffic has grown phenomenally.

That torrid growth continues, despite reports to the contrary based on ComScore data, according to John Adams of Twitter’s operations team, who spoke this morning at the O’Reilly Velocity Conference in San Jose. “There are a lot of reports that our growth is slowing down,” said Adams. “I can’t say what the real numbers are. But it’s just not slowing down at all. All that traffic has led to an insane amount of pain.”

Measuring and analyzing performance data has been the primary weapon in Twitter’s ongoing effort to vanquish the “Fail Whale” - the downtime mascot that appears whenever Twitter is unavailable.

“You really want to instrument everything you have,” Adams told an audience of 700 operations professionals. “The best thing you can do is have more information about your system. We’ve built a process around using these metrics to make decisions. We use science. The way we find the weakest point in our infrastructure is by collecting metrics and making graphs out of them.”

Chatsworth Products Acquires Epicenter

Data center equipment maker Chatsworth Products, Inc. (CPI) today beefed up its portfolio of data center management tools with the acquisition of Epicenter Inc., a Westborough, Mass. company specializing in KVM (keyboard-video-mouse) management solutions.  

The deal continues a series of deals in which vendors targeting the data center have focused on acquiring tools and software to help customers manage their increasingly complex data center environments. Epicenter’s systems measure data center performance metrics such as compute per watt, and provide centralized cross-platform management of servers, network and storage.

Epicenter was founded in 1988, and its technology has been installed at more than 1,000 enterprises in 15 countries. The company markets its Centerline and Centerpoint solutions through distributors and OEM partners.

Interop Launches: Avocent, 3tera, Apcon, Aten

It’s been a busy week for new product announcements at the Interop show in Las Vegas. Here’s a roundup:

• Avocent Corporation (AVCT) announced Avocent MergePoint Infrastructure Explorer, a software platform that integrates resource management (heat, cooling, power, and space) with asset management, providing a centralized automated view of these resources from a single location and with a single product. “It’s clear from talking to customers that the traditional domains of facilities and IT are merging,” said Avocent CEO, Michael Borman. “Companies want to view and manage their environment using one management tool. Avocent MergePoint Infrastructure Explorer accomplishes these goals, effectively eliminating the need for multiple point products currently available on the market.”
• 3Tera, Inc., the leading innovator of cloud computing technology and utility computing services, announce the introduction of the 3Tera AppStore, a  marketplace where enterprise users and software vendors can exchange cloud components on a pay-per-use basis. The AppStore catalog spans all types of datacenter infrastructure elements and complete applications, from networking components (such as load balancers, firewalls, routers, traffic managers, content switches), server components (such as Web servers, directory servers, databases, application servers) and storage solutions, as well as application software stacks, management and monitoring tools. ” We’ve been enabling Cloud Computing for the past three years,” said Barry Lynn , Chairman and CEO, 3Tera, Inc. “With the introduction of 3Tera’s AppStore, we’re enabling a community of software providers to make their products easily available in the Cloud and accessible on-demand.”

Integrated Approach to Energy Management

In the quest to more effectively manage the IT infrastructure, many organizations are stymied by a disconnect between IT and Facilities. The two groups rarely collaborate and typically report into different parts of the organization. Apart, neither group can optimize energy consumption and system availability across the big picture. The data center could be consuming more energy than necessary. Essential IT applications could be at risk from infrastructure weaknesses. Inefficiencies could force premature construction of additional power and cooling infrastructure.

If IT and Facilities could work collaboratively, organizations can operate more efficiently and effectively while still meeting their business objectives. This white paper from Eaton describes how a joint monitoring and management solution links IT assets, the data center infrastructure and Facilities assets into a holistic perspective aligned with business processes.

Click here for a copy of this Eaton white paper (free to registered members).

Pandemic Planning and the Data Center

Data center professionals deal with risk assessment and management every day. Nonetheless, pandemics present different challenges than other disaster scenarios, particularly in their potential to affect operations and staffing across a wide range of locations. With the World Health Organization now calling on countries and organizations to activate pandemic preparation plans, it seems prudent to highlight pandemic resources for IT and data center managers:

• At ComputerWorld, Scott McPherson highlights several pandemic presentations he created for the Florida  CIO Council’s pandemic web site, including a pandemic planning checklist for IT departments and data centers. Key issues include how to adapt data center operations in the event staffers are unable to work due to illness or quarantine, including a prioritization plan for applications and services to determine which are essential if you run out of staff or resources.
• The National Association of State CIOs has a pandemic planning document for state IT departments, titled “Where’s My Staff?” (PDF) which covers many similar issues. It notes the importance of remote access and suggests priority IP access for critical employees.
• A 2007 article from Processor highlights additional issues, including the potential to relocate staff out of areas of active pandemic risk, and the need to examine supplier and vendor relationships for potential impact.
• Pandemic planning was discussed at the 2006 Gartner Data Center Conference, which highlighted the potential impact on companies who have outsourced IT operations to countries that may be affected by a pandemic. 
• For a big-picture overview of potential impacts, see the U.S. Department of Homeland Security publication on Pandemic Influenza  Preparedness, Response and Recovery for Critical Infrastructure (PDF, 84 pages). 

What other issues are helpful to keep in mind for IT and data center staff? If you have expertise, advice or useful links, please share them in the comments.

Tier1: Higher Prices Ahead for Data Centers

Will 2009 be a huge year for enterprise data center outsourcing? It will if enterprise companies understand the pricing outlook for the data center market, according to Dan Golding, vice president and research director of Tier1 Research. Demand for data center services has declined only slightly, Golding says, while the amount of new supply has been constrained by the credit crunch. The tough economy has led many many providers to hold pricing steady, but that trend won’t hold as the supply-demand imbalance continues to deteriorate in major data center markets, particularly if the economy improves.

“I think we’re going to see data center prices increasing significantly in the third and fourth quarter,” Golding said Tuesday in his keynote at Tier1’s first Datacenter Transformation Summit in Herndon, Va. “We think we’re going to see 7 to 10 percent price increases annually until credit eases, plus another 18 months (to fund and build new data centers). This is good for providers but not for enterprises. Now is the time for enterprises to lock in. There are lots of things you can get a discount on by waiting. Data centers are not one of them.”

The impact of the credit crunch on supply emerged as a challenge in early 2008, even before the financial crisis worsened in September. Tier1 has been predicting that a shortage of quality data center space could lead to higher colocation prices, although pricing impacts will vary among market segments and geography, Golding said. “There are specific markets where you can get deals as providers seek to meet bank covenants,” he said, referring to loan agreements in which repayment terms are linked to leasing success.

Lee Technologies to Manage DRT Facilities

Lee Technologies has been selected by Digital Realty Trust, Inc. (DLR) to operate 10 of its data center facilities around the United States, the companies said today. Lee Technologies, which supplies staff to manage data center infrastructure, said it needed just a month to transition a team of 35 facility technicians to support the 10 Digital Realty Trust data centers in Phoenix, Chicago, St. Louis, Dallas and Houston.

“Lee Technologies quickly demonstrated its mission-critical expertise after successfully providing facility operations at our Paris and Dublin data centers, and we have full confidence they will deliver the same level of performance across these U.S. multi-customer facilities,” said Chris Crosby, Senior Vice President of Digital Realty Trust. “With Lee Technologies’ facility technicians at 10 of our primary mission-critical, multi-customer facilities will enable us to continue to deliver a five 9’s level of reliability in our Turn-Key Datacenter facilities as we have for the past four years.”

Coleman: Cassatt ‘Close to the End’

Data center management software firm Cassatt is “close to the end” and likely to file for bankruptcy after failing to find an acquirer, CEO Bill Coleman tells Forbes. Cassatt initially sought to gain a foothold in energy management, positioning its software as a tool to help data center managers automate the cycling down of idle servers. Cassatt also focused on management of private clouds, but has failed to gain traction in an increasingly crowded market for cloud solutions. Here’s an excerpt from the Forbes article.

Coleman and others have been quietly shopping Cassatt around for several months, often nearing a deal only to see it collapse. Coleman would not say which companies were approached, but others involved with the talks said the parties included all its blue chip competitors. Google and Amazon, the source said, cut off talks early on, contending they do not want to get into the corporate computing business at present. “I have talked with about a dozen companies, all the usual suspects,” says Coleman. “There are one or two possible buyers, and a couple of flickers of interest, but pretty soon I have to think about what’s best for my shareholders.” Cassatt’s software, which Coleman said repeatedly proved its management cost-saving capability, would probably be sold in a bankruptcy to one or another major firm.

Investors in Cassatt included Warburg Pincus and New Enterprise Associates. See Forbes for more details.