The NASDAQ OMX Group today introduced the Super Cab, a cabinet that vents the hot exhaust air from servers into a custom cabinet chimney, allowing it to cool customer workloads requiring up to 17 kilowatts of power within the NASDAQ OMX Data Centert. The solution also reduces cabinet-to-cabinet connectivity latency and maintains reliability with fully redundant power capacity.

The NASDAQ is among the exchanges offering colocation services to trading customers who want to house their servers as close as possible to exchange’s “matching engines” – the servers that match buy and sell orders and provide updated pricing information. The need to execute a trade a microsecond ahead of rivals has created an arms race in ultra-low latency colocation, and is reshaping the playing field in the trading industry.

The NASDAQ OMX Data Center is located in New Jersey, and is one of the largest U.S. trading hubs in the world, hosting both equities and options markets and providing access to major industry market data feeds and multiple telecommunications providers.

NASDAQ OMX colocation services are available to all market participants and provides customers the opportunity to place their own trading systems within the NASDAQ OMX Data Center with equal distance to the matching engine. Access Services also offers various levels of power, access and bandwidth within the NASDAQ OMX Data Center to give customers options according to their individual needs.

NASDAQ OMX also announced the coming availability of a new high-speed network offering, 40G Client Connectivity, which will allow customers to access all of the exchange group’s U.S. markets through a single ultra low latency connection. Network latency on 40 GB is expected to be reduced by an average of 7 microseconds roundtrip compared to the current 10 GB connectivity option available to colocation clients. In addition, customers can use 40G Client Connectivity to access all data feeds offered in the NASDAQ OMX Data Center, including NASDAQ OMX market data and many other U.S. exchange feeds and consolidated feeds.

“NASDAQ OMX is pleased to be the first exchange to offer these innovative co-location solutions,” said Stacie Swanstrom, Vice President of Transaction Services at NASDAQ OMX. “Super Cab and 40G Client Connectivity add to the most advanced offerings available at the NASDAQ OMX Data Center, providing the highest tiers in a suite of services for power, access and bandwidth.”

The long main hallway of the NYSE Euronext data center provides a sense of the immense scale of the 400,000 square foot facility in New Jersey. (Photo: Rich Miller)

The New York Stock Exchange says it expects a normal trading day after a small fire Sunday at its data center in Mahwah, New Jersey, which supports computer systems that are critical to the U.S. financial markets. The NYSE says the incident briefly interrupted connectivity for some customers, but should have no impact on its trading operations for Monday.

“On Sunday, there was an isolated electrical fire that was quickly extinguished within a single computer cabinet at our Mahwah data center,” the NYSE said in a statement. “The incident, which resulted in no injuries, affected communications connectivity to 58 customers who have been notified that we are testing all systems and expect completely normal operations for Monday’s market open.”

The 400,000 square foot data center in Mahwah serves as the nerve center for the NYSE”s electronic trading operations. The facility is staffed around the clock by employees trained to respond to electrical fires and other emergencies. Like most major data centers, the NYSE facility has systems that monitor temperature in data halls and provide alerts in the event of sudden changes.  Data centers also are equipped with sensitive fire detection systems triggered by the presence of smoke or heat, which tie into fire suppression systems.

The NYSE didn’t provide details on how the fire was extinguished, but it appears that the event was quickly contained.

“We are grateful for the quick and thorough response by the Mahwah Fire and Police Departments as well as our Mahwah data center staff,” the exchange said.

Fires within data centers can cause major downtime for customers, as seen in 2009 outages at Fisher Plaza and 151 Front Street. That’s not always the case, as seen in a 2010 incident at Terremark’s NAP of the Capital Region in Virgina in which a fire broke out in one of the data center electrical rooms, but the facility remained online throughout the entire event,  with no downtime for customers.

Fortunately, the event at the NYSE was small. Another factor protecting modern data center facilities is the use of “pod” designs that segment the power and cooling systems to limit the impact of electrical events. The Mahwah facility includes three data center pods. Each is approximately 20,000 square feet, and has dedicated power and cooling systems so that a failure in one pod won’t affect operations of the other data halls.

NASDAQ OMX has confirmed that its Stockholm premises had been visited by regulators and said it is cooperating with the inquiry. It apparently is not the only company that has been contacted.

“The Swedish Competition Authority can confirm that it had cause to make unannounced visits to companies in the financial sector,” the regulator said in a statement. “These visits comprise a step in the search for evidence that may support suspicions of possible anti-competitive cooperation or abuse of dominant position.”

The case stems from an incident last fall in which trading group Burgundy complained to the Swedish Competition Authority, saying it was refused access to a Verizon colocation center in Lunda, Sweden used by Nasdaq OMX and many Swedish banks and brokers.

Burgundy and its technology provider Cinnober said Verizon Business reneged on an agreement for Cinnober to move Burgundy’s matching engines to the Lunda data center (see coverage at Low-Latency.com and Waters Technology). As a multiateral trading facility (MTF), Burgundy is a rival to the NASDAQ OMX exchanges in the Nordic countries. Burgundy wound up moving its matching engines to an Interxion data center about five miles away.

Verizon Business has not offered public comment on the complaint. But Olof Neiglick, the chief executive of Burgundy, says his complaint has relevance for other markets where data centers are emerging as high-speed trading hubs. “This goes way beyond Sweden because it undermines the whole idea of competition between exchanges if someone can dictate who gets into which data center,” Neiglick told the FT.

The long main hallway of the NYSE Euronext data center provides a sense of the immense scale of the 400,000 square foot facility in New Jersey. (Photo: Rich Miller)

There are few data centers in the world that are more difficult to get into than the new NYSE Euronext facility in Mahwah, New Jersey.  The huge building features extremely strong access control, a substantial security perimeter, and is built to survive … well, just about anything.

“It is more robust than your typical structure,” said Steve Rubinow, Chief Information Office at NYSE Euronext, who said the data center “can withstand levels of punishment – both man-made and natural – that other facilities might not withstand.”

The 400,000 square foot data center has been engineered for security, speed and reliability to support its mission as the nerve center for the NYSE”s electronic trading operations. Starting tomorrow it will add a new role: powering the financial industry’s first cloud computing platform, which will let traders lease computing power at the new data center.

Technology Drives Trading Transition

The Mahwah data center, along with a similar facility outside London, provides the horsepower driving a major transition for NYSE Euronext. As the financial markets are being transformed by technology and global trading networks, data centers have become the industry’s new trading floors. NYSE Technologies, the IT arm of NYSE Euronext, set out to build a facility with extraordinary resiliency.

“Every time we conduct a tour, even the hardened IT guys who seen a lot of data centers universally come away impressed,” said Rubinow.

NYSE Technologies  is proud of its facility, but cautious about revealing too many details of its security and reliability features. That includes the power feeds from utility providers, which provide particularly robust support. “On utility related matters, we benefit from features that are rarely found at any site currently operating,” the NYSE said. The utility feeds are backed up by banks of generators and a large supply of on-site diesel fuel.

High Redundancy in Power, Cooling Systems

All of the power and mechanical systems in Mahwah are engineered to at least 2N reliability, with backup equipment available for each piece of critical infrastructure. The mechanical and electric infrastructure occupies nearly three quarters of the building, with 100,000 square feet of space for server rooms. Each data center pod is approximately 20,000 square feet, and has dedicated power and cooling systems so that a failure in one pod won’t affect operations of the other data halls.

NYSE Technologies has completed three pods thus far, creating 60,000 square feet of raised floor space to house servers and storage for the exchange, its colocation clients and its new cloud service, the Capital Markets Community Platform. The Mahwah facility has space for two more pods, which will be completed as the existing space nears capacity.

“We made sure the power and cooling systems for each pod were isolated,” said Rubinow. “When we talk about having independent pods, you have to figure out the network and how we’ll keep working in the event of a single pod failure.”

The Need for Speed

NYSE Technologies’ cutting edge network is a key selling point for its colocation and cloud computing customers. In seeking to create one of the world’s fastest low-latency network, NYSE Technologies worked closely with networking vendors, including Juniper Networks. By collapsing the multiple switching layers present in traditional network architectures, the NYSE Euronext network design requires fewer devices and interconnections, leading to improved efficiencies in space, power, cooling and management.

“Performance is key,” Rubinow said. “We wanted to know exactly how long it took to get (data) through their equipment. The other issue is how quickly it can scale. We have to scale it in a flexible fashion.”

For the NYSE, scaling isn’t simply a matter of how many customers it hosts. The network must be able to scale to handle surges in trading volume on its exchanges, providing a mission-critical bursting challenge.

A “Science” to Scalability

“There’s a science to that, and also an art,” said Rubinow. “We can scale for a new customer, and that’s predictable and pretty easy to manage. The harder part is when our large customers increase their trading activity, because they can begin to fill our pipes very quickly. If there’s an unusual event in the world, their models generate lots and lots of traffic. The key question is how much buffer you put in (for traffic spikes).”

There are also network design challenges related to the NYSE’s proximity hosting services for customers who prize high-speed connectivity. A key goal is ensuring that the colocation environment maintains a level playing field.

“It’s very important that our customers are equal citizens,” said Rubinow. “We design the network so that it provides all of our customers the same access. On the things that make a difference to traders, we make sure the paths are operationally the same.”

Site Selection and Expansion

After years of using or leasing third-party facilities, NYSE Euronext didn’t set out to build a greenfield data center.

“When we started the project, we were looking for a space where we could continue to grow and we could consolidate severs,” said Rubinow. “We looked all up and down the eastern seaboard for existing facilities. After looking for a long time, we didn’t find a physical facility that met our needs. Our requirements eliminate a lot of existing data centers and geographies pretty quickly. If you believe that the attributes of the facility add value, the best way to do it is to control your fate.”

Like many data center operators, NYSE Euronext carefully weighed how much to capacity it needed to build. That meant estimating the growth of the market for low-latency colocation services and the amount of space that might be needed to support the exchange’s cloud computing ambitions.

“The data center decision is typically a 15 to 20 year decision,” said Rubinow. “You don’t want to build too much. You might have the kind of growth that (forces expansion), but that’s a high-quality problem. We plan on a 15 to 20 year life for the data centers. In both locations, we have an ability to build little more space if we need it.”

Although most financial institutions have a disaster recovery and business continuity plan in place, vulnerabilities tend to creep in over time. Many of these institutions operate with lean IT staffs and the focus is on keeping the systems running day to day. As the bank grows, IT systems are changed and expanded ad hoc, based more on operational needs than along the lines of a strategic plan. Acquisitions and mergers can add even more nonstandard technology to the mix. This results in an accumulation of equipment and technologies that is hard to manage, prone to failures, riddled with security problems, expensive to operate and vulnerable to disruption. Without realizing it, these organizations are operating in a house of cards. The crash of a single server may create a domino effect that brings the entire network down. The failure of a storage unit that was overlooked in a backup could mean the loss of business-critical data.

It doesn’t have to be this way. You can significantly improve your disaster recovery capability without a huge investment. This white paper show how by using the right technology and best practices you can significantly reduce the cost of preparing for and recovering from a disruption or a disaster. Click here to get this white paper.

As anticipated, Nasdaq and ICE today announced a joint bid for NYSE, valued at around $11.3 billion, topping the value of NYSE Euronext’s planned merger with Deutsche Börse AG. In a conference call, Nasdaq CEO Robert Greifeld said the deal would involve a major consolidation of IT infrastructure.

“One data center will be closed, but we don’t know which one would be the best candidate right now,” Greifeld said.

New Jersey A Consolidation Candidate?
Which region may be affected? The most obvious candidate would be New Jersey, where both exchanges operate major data centers.

NYSE Euronext recently opened a new 400,000 square foot data center in Mahwah, New Jersey to serve as the home to the exchange’s new matching engines – servers that match buy and sell orders and provide updated pricing information.The new data center features colocation space for trading firms seeking high-speed access to the matching engines. NYSE Euronext says it has sold out all the available colocation space in its first phase at Mahwah – reported to be at least two 20,000 square foot pods. Three additional pods are planned as expansion space, for a total of 100,000 square feet of colocation space.

Nasdaq OMX Group doesn’t own any data centers, but leases a number of facilities. The primary data center for Nasdaq OMX Group (NDAQ) is housed in a Verizon Business facility in Carteret, New Jersey, while its backup facility is operated out of a Verizon Business site in northern Virginia. In Dec. 2009 NASDAQ OMX expanded its colocation agreement with Verizon Business to offer additional colocation space to trading firms engaged in high-frequency trading.

NASDAQ also uses the Amazon Web Services cloud computing platform to data on historic trades and lets investors analyze pricing in relation to news events and earnings calls to gauge the market response.  Robert Waghorne, Senior Vice President of European Markets Technology at NASDAQ OMX Group, recently discussed the company’s  operations at the DataCenterDynamics NY conference.

Will a consolidation decision be driven by the quality of the infrastructure, or whether a site is owned or leased? How will the global footprints of the two companies factor into any decisions?  The Nasdaq OMX has many hurdles to overcome before we’ll see any data center consolidation, but this is a clearly a story to watch

TIBCO says its new offering can push new frontiers in high speed trading, where minute advantages in message delivery translate into a significant competitive advantage.

“Markets have an insatiable need for speed,” said Roy Schulte, Distinguished Analyst and Vice President, Gartner. “Low latency messaging technology is critical to any firm’s strategy to dominate today’s extreme-high-speed trading floors. Firms compete head-to-head on speed of execution, message volumes are exploding and integration needs are getting more and more complex. Firms are looking at messaging solutions to provide the lowest possible latency to beat the competition – but also to provide key capabilities so that they are agile enough to sustain that advantage.”

When Nanoseconds Count

In testing, TIBCO said FTL displayed average application latency in test conditions of 384 nanoseconds for intra-host communication and 3.1 microseconds for inter-host communication. The company says the product, which was first previewed in May 2010, provides traders with the opportunity to better exploit faster networks and hardware.

“TIBCO’s new messaging technology is optimized for both intra-host and inter-host messaging, something that is critical in leading these paradigm shifts in computing,” said Kirk Skaugen, Vice President and General Manager of the Intel Data Center Group. “As companies move more of their processing to multi-core machines, TIBCO FTL software running on Intel Xeon platforms will deliver an order of magnitude speed increase in areas where every microsecond—no every nanosecond—counts. We’re excited at the results of this engineering collaboration with TIBCO. We have created technology that is revolutionizing high performance messaging capabilities.”

TIBCO Software Inc. (TIBX) provides infrastructure software for companies to use on-premise or as part of cloud computing environments. It has more than 4,000 customers worldwide.

Arista’s sub 500 Nanosecond switch. Arista Networks announced the 7124SX, touting it as the “fastest switch in the world,” with standard SFP+ interfaces. With sub-500 nanosecond latency, the Arista 7124SX is ideal for financial trading and ultra low latency applications as a leaf or access top-of-rack 10Gigabit Ethernet switch.  The new switch is a wire-speed, 24 port Layer 2/3 switch and supports 100Mb, 1GbE and 10GbE SFP+ based optics and cables in one rack unit. The switch not only represents a highly efficient and compact design, but improves latency by 20 percent – over 100 nanoseconds, according to Arista. The product is available immediately and starts at $12,995.  Arista also introduced Latency Analyzer (LANZ), a new software capability to monitor and analyze network performance, generate early warning of pending congestion events, and track sources of bottlenecks. Arista will be at the GigaOm Big Data conference in New York next week.

XO unveils cloud security solution.  XO Communications (XOHO) announced the launch of its new cloud-based security solution that offers businesses the ability to more quickly and cost-effectively deploy and manage comprehensive network security capabilities across their enterprise networks. XO Enterprise Cloud Security delivers network security capabilities delivered through the XO network as a security as a service offering. Capabilities able to be delivered include network based firewall, intrusion detection and prevention, secure web gateway for web and content filtering, and secure remote access to corporate network. “XO Enterprise Cloud Security significantly expands our security services portfolio and represents the continuing expansion of our cloud-based services delivery strategy,” said Mike Toplisek, chief marketing officer for XO Business Services at XO Communications. “By offering network security as a service, mid-sized businesses and larger enterprises can simplify the management of their security requirements and leverage our expertise to provide a unified threat management solution across their enterprise.”

Gartner examines key data center issues. At the Infrastructure, Operations and Data Center Summit in Sydney Tuesday Gartner highlighted four key forces operating in the market today that could result in data center space requirements that will shrink dramatically before the decade is out. “In the world of IT, everything has cascade effects, and in data centers the traditional methods of design no longer work without understanding the outside forces that will have an impact on data center costs, size and longevity,” said David Cappuccio,managing vice president and chief of research for infrastructure at Gartner. “However, these very forces can actually work in your favor, providing the means to apply innovative designs, reduce capital costs and operating costs, increase long term scale, and keep up with the business.” The recommendations for designing and operating a data center of the future were to focus on smarter designs, the ‘greening of IT’, conquering desnsity, and cloud computing. Additional information on these recommendations and the topic can be found in the Gartner report “Shrinking Data Centers: Your Next Data Center Will Be Smaller Than You Think.”

As a pioneer in electronic trading, Nasdaq OMX Group has been a technology leader as U.S. stock exchange action has shifted from bourse floors to data centers. The  explosive growth of high frequency trading has now sparked a technology arms race on Wall Street, in which technology platforms have become the crown jewels in a flurry of mergers and acquisitions. As trading titans seek to assemble global platforms, Nasdaq is working on several fronts to retain a leadership position.

In the boardroom, Nasdaq is contemplating a hostile takeover bid for NYSE Euronext, according to reports this morning in the New York Times and Wall Street Journal.  The bid is aimed at averting a planned merger between the NYSE and Deutsche Borse that would merge leading U.S. and European platforms. According to reports,  the bid would rely upon Nasdaq OMX’s ability to borrow $5 billion and work out a deal with InterContinental Exchange (ICE) to buy some NYSE Euronext assets.

In the data center, Nasdaq has been constantly working to provide technology to support faster and cheaper trades for the hedge funds and traders using its high speed trading platform, according to Robert Waghorne, Senior Vice President of European Markets Technology at NASDAQ OMX Group, who discussed the company’s  operations last week at the DataCenterDynamics New York conference.

Technology as the Core Business Engine

“We really are a technology company,” said Waghorne. “It’s the core engine for all of our business. We can trade anything, anywhere on the planet.”

In addition to its familiar NASDAQ electronic exchange in the U.S., the company’s technology and software powers 70 exchanges in 50 countries around the world, including everything from electricity markets in the UK to stock exchanges in Australia.

In the U.S. Waghorne says the Nasdaq platform can handle more than 1 million messages a second at an average speed under 100 microseconds. “The exchanges need to be as fast as possible,” said Waghorne. “It’s a baseline to be in this industry.”

The need for speed is a key element in the growth of high speed trading, in which traders use algorithms to identify and capture opportunities for profitable trades. This approach relies upon speed in both directions – obtaining price information and executing trades. The growth of competing electronic platforms and exchanges has placed a premium on transaction cost as well as execution speed.

Cost Pressures Drive Efficiency, Consolidation

“The cost pressures on exchanges are enormous, so we need to be hyper-efficient,” said Waghorne. “The (transaction) costs that customers are expecting are decreasing, so you need massive scale to make this work. This is what is driving consolidation in the industry. Technology is a major factor in exchange consolidation.”

In seeking to reduce transaction costs, Nasdaq has looked to its IT department to deliver savings. “Data center consolidation and platform consolidation has been central for us,” said Waghorne. “We’ve definitely been pushing into multi-core (processors and servers), and we’ve made some significant moves with virtualization that have driven cost reductions.We have reduced technology costs 40 percent from 2006 to 2010 while increasing capacity. In 2006, it took about a millisecond to conduct a transaction. Now it’s under 100 microseconds.”

All the while, volume has been surging. In 2006 Nasdaq’s systems handled 53,000 messages per second. Today it manages more than 500,000 messages per second, on average. “Our throughput requirements have gone through the roof,” said Waghorne.

Active Colocation Business

That has made Nasdaq’s data centers an appealing environment for proximity trading, “We have been a pioneer in offering colocation services allowing our users to put servers in our data centers,” said Waghorne. “Not only are we selling the data center space, but we’re now offering a more significant suite of tools. It’s a growth area for us.”

Nasdaq doesn’t own any data centers, but leases a number of facilities. The primary data center for Nasdaq OMX Group (NDAQ) is housed in a Verizon Business facility in New Jersey, while its backup facility is operated out of a Verizon Business site in northern Virginia. In Dec. 2009 NASDAQ OMX expanded its colocation agreement with Verizon Business to offer additional colocation space to trading firms engaged in high-frequency trading.

Waghorne says that the surge in electronic trading has underscored the wisdom of Nasdaq’s approach to its business. “We look at ourselves as a pioneer in electronic trading,” he said. “We’ve always been an electronic exchange.”