Cisco space router operates in orbit. On Monday Cisco announced that their Internet Routing in Space (IRIS) technology has achieved a major milestone with the successful in-orbit test of the Cisco IOS software’s networking capabilities and the company’s on-board router. The technology was aboard an Intelsat IS-14 commercial GEO satellite and launched November 23, 2009. The Cisco IRIS technology is a program to build a radiation-tolerant IP router for satellite and related spacecraft. Cisco vice president for the Global Government Solutions Group Steven Boutelle said “this milestone is another step in our strategy to expand borderless networks into space and redefine how satellite communications are delivered. This technology can help transform satellite communications around the world by reducing latency and increasing the efficiency.” Managed by Cisco and Intelsat the IRIS program is a Department of Defense Joint Capability Technology Demonstration (JCTD) and the payload will convert to commercial use following a three month JCTD in April 2010.

AboveNet connects all 20 Equinix U.S. data centers. Equinix (EQIX) and AboveNet (ABVT) announced that AboveNet has deployed operations to the Equinix New York-1 data center, marking the company’s deployment to all 20 of Equinix’s IBX centers in the U.S.  AboveNet offers managed services, Ethernet, metro networks and a global Tier 1 optical IP network.  Making use of the Equinix global services delivery platform, AboveNet will enhance peering capabilities with other Tier 1 networks.  Equinix chief marketing officer Jarrett Appleby said “our partnership with AboveNet offers a strong Tier 1 network and high bandwidth connectivity solution options for the community of IBX participants while offering AboveNet significant new revenue opportunities and a national infrastructure for Tier 1 peering and future carrier Ethernet needs.”  Equinix acquisition target Switch and Data announced that Lexent Metro Connect, a dark fiber network provider in New York, has become part of the Switch and Data GeoReach program.  Lexent Metro Connect has also completed construction of a low-latency dark fiber network route to Switch and Data’s North Bergen site located in New Jersey.  The GeoReach program is a select group of providers who have engineered their networks to meet the needs of the high-efficiency trading community.

Huawei to deploy DWDM platform for GlobeNet. Global telecommunications solution provider Huawei was selected to deploy a next generation DWDM (Dense Wavelength Division Multiplexing) optical platform for network transport services company GlobeNet. The Huawei platform will connect GlobeNet’s cable landing station in southern New Jersey to its Point of Presence sites in New York city via two diverse paths. The diverse paths will allow GlobeNet to deliver services to their other sites in Brazil, Venezuela, Bermuda, Florida and others. The network solution provided by Huawei will support 10Gb per second, 40GB per second and future 100Gb per second wavelengths to competitively deliver advanced, high bandwidth services.

For additional information, check out our SuperNAP Channel and Sun Microsystems Channel. For additional video, check out our DCK video archive and the Data Center Videos channel on YouTube.

Switch Communications CEO Rob Roy with one of the WDMD custom cooling units at the Las Vegas SuperNAP during a tour last year.

LAS VEGAS -High on a narrow catwalk alongside the massive cooling units at the SuperNAP, a security guard stops to open one of the four doors lining the side of the unit. As the door opens, a powerful blast of air streams out. “You have to remember to hold on to keep from getting blown off,” said Melissa Young, the Executive VP of Sales Engineering at the SuperNAP, a 407,000 square foot data center facility built by Switch Communications.

The cooling unit is a WDMD – short for Wattage Density Modular Design – a custom-built unit housed outside the data center that can automatically switch between four different cooling options to deliver the most efficient cooling for current conditions. Young says the WDMDs are “built by Switch, for Switch” and not available from any vendor.

The units are part of the customized power and cooling infrastructure at the SuperNAP, where Switch also builds its own power distribution units (PDUs) and remote power panels. Young says the SuperNAP’s generators are also customized to Switch’s specifications by  Detroit Diesel.  

At a time when many large data center builders are focused on the industrialization of data center construction using standardization and bulk purchasing from vendors, Switch is charting a different path, building custom equipment to fit its vision for high-density data centers supporting power loads of 1,500 watts a square foot and beyond. It’s a philosophy also seen at Google, which builds its own servers, containers and networking gear.

Growth Beyond Las Vegas?
Can this model scale beyond Switch’s hugely successful data center operation in Las Vegas? Young says Switch is scouting prospective data center sites in other markets, but has yet to decide whether to pursue projects outside Las Vegas.

“We are talking to a number of people about either having us build data centers for them or licensing our technology,” said Young.  

Much of the company’s expertise in extreme-density infrastructure could work in other markets and facilities. But the secret sauce supporting the Switch Supernap goes beyond custom infrastructure. The desert climate and unusually rich connectivity and bandwidth economics have also been huge factors in Switch’s success, and are less portable to other venues.

Fitting Out the SuperNAP 
Bur first, there’s Vegas and the SuperNAP. With the first 45,000 square foot pod nearing capacity, Switch has completed the fit-out of a second pod and says it expects a significant portion of the new space to be filled by large requirements from existing tenants. “We didn’t figure we’d need Sector 2 until early next year,” Young says.

The building includes six of these pods. “Once this building is full of gear, it will be the most densely-packed data center in the world,” she said. As the SuperNAP nears capacity,  Switch plans to build two more facilities on adjacent property.

More Power, Fewer Racks
Young says the SuperNAP’s ability to pack larger workloads into fewer racks offers a compelling value for customers with “Internet-scale” large workloads. “The power and cooling configuration really does change the equation for our customers,” said Young. “The density allows you fewer racks, and companies usually do a (hardware) refresh to take advantage of that.”

Young said many customers find the high-density installations also save on cabling, since there are fewer racks and no need for additional spacing between racks to avoid hot spots.

She said most SuperNAP customers are running equipment at between 8 kilowatts and 17 kilowatts a rack, with one customer at 24 kilowatts. With 100 megawatts of power for the facility, Young said Switch expects to ultimately be able to support 7,000 racks, although that number might edge lower if rack power densities increased. “100 megawatts is a lot of power, but it’s a finite number,” she said. “It’s not a tether to the sun.”

RELATED STORIES:

• A Look Inside the Vegas SuperNAP
• 1,500 Watts A Foot? A Look at TSCIF
• Video: Cooling the SuperNAP
• The Vegas SuperNAP: A Data Center Revolution?

“We now have thousands of cores at the SuperNAP,” Papadopoulos said in this morning’s keynote address at AFCOM’s Data Center World conference at the Paris Hotel in Las Vegas. “It’s a really fascinating facility.”

The hosting arrangement extends the relationship between Sun and Switch Communications, which is already hosting Sun’s Network.com operation in a high-density section of SwitchNAP 4 in Las Vegas known as a T-SCIF heat management system (short for Thermal Separate Compartment in Facility). The T-SCIF uses containment systems to fully separate the hot and cold aisles, allowing the Network.com racks to run at 1,500 watts a square foot. See this video for a closer look at the Network.com T-SCIF installation at Switch.

Sun’s Network.com was an early entry in the online utility computing arena, but struggled to gain traction and is now “in transition” and closed to new users. Sun is developing a next-generation cloud offering, but has yet to provide details of the new service.

Papadopoulos noted that Sun has been developing a platform known as Project Caroline, and recently acquired Q-Layer, a Belgian provider that automates the deployment of both public and private clouds. Sun says Q-layer’s technology will help users instantly provision servers, storage, bandwidth and applications.

Papadopoulos spoke at length about the importance of developers in the cloud ecosystem, noting that many developers are using Amazon Web Services to test-drive new web applications, at least partly out of frustration with delays in provisioning test environments through in-house IT operations.

“I give Amazon credit for having broken through, either through their brilliance or serendipity, and creating something that really works,” said Papadopoulos. “It’s about making things easy and accessible for the developer.

“We always think of the cloud in terms of developers,” said Papadopoulous, who added that the best commercial opportunities may be in helping companies build their own private cloud infrastructure and leverage public clouds in a “hybrid” cloud format. “Where you make money, I think, is in the hybrid cloud. Most enterprises are going to have their own private cloud.”

Papdopoulos said Sun will use the SuperNAP for “aspects of our cloud computing operation,” indicating that Sun may use its own facilities as well.

The 407,000 square foot SuperNAP, which opened its doors in September, will hold up to 7,000 cabinets when it is completely built out. Switch CEO and co-founder Rob Roy reported that Switch has now sold more than 1,100 cabinets. Other sources say that Sun made an early commitment to take space at the facility, perhaps as as many as 200 cabinets.

Papadopoulos praised the SuperNAP’s approach to high density computing,  using part of his keynote to present a video of the facility’s cooling system, which foregoes computer coom air conditioners (CRACs), venting hot exhaust air into a large ceiling plenum that returns the air to custom cooling units located outside the building, known as WDMD (short for Wattage Density Modular Design). See our video overview of the WDMD cooling units.

We had an inside look at the SuperNAP this summer, and shot video of our tour of the massive facility while it was under construction, as well as two of the key technologies featured at the SuperNAP: the T-SCIF containment system and WDMD cooling units. Switch Communications and APC by Schneider (a major vendor for Switch’s facilities) recently put together a video offering a look inside the completed SuperNAP facility. This video runs about 9 minutes.

For more news about the SuperNAP, visit our Switch Communications channel. For additional video, check out our DCK video archive and the Data Center Videos channel on YouTube.

When the first phase of the SuperNAP opens on Sept. 1, it will be one of the world’s most unique data centers, with the ability to cool racks exceeding 20kW of power load. When the facility is completed, it will cost more than $300 million and be able to host 7,000 customer servers.

The SuperNAP will have no raised floor, no computer room air conditioning units (CRACs) inside the data center, and no use of liquid cooling – in fact, virtually no water in the entire building. The massive facility is the ultimate expression of an alternate view of high-density data center design, formulated by Switch Communications CEO and co-founder Rob Roy.

“My feeling is that when people see this, they’ll say that this is the answer going forward,” said Roy. “With our new design, we may be able to get to 2,000 watts per square foot. We’re very excited about what we’re doing.”

Data Center Knowledge recently got an inside look at Switch Communications’ Las Vegas operation, including the ultra-high density hosting area of its existing SwitchNAP facilities, where several prominent Internet companies are running banks of racks at 1,500 watts a square foot using Switch’s high-density T-SCIF heat management system (short for Thermal Separate Compartment in Facility). See this video for a look inside a T-SCIF for Sun Microsystems, which hosts its Network.com utility computing platform at Switch.

We also had a tour of the SuperNAP facility, which was in the late stages of construction, and got a look at the custom central cooling units that Roy says will take air cooling to unprecedented levels of efficiency and flexibility. The units, known as WDMDs (for Wattage, Density, Modular Design) have four coils to allow different approaches to cooling in different conditions. See videos of our walk-through at the SuperNAP and a closer look at the WDMD cooling units.

Until very recently, Switch has flown under the radar in the data center industry. After starting with a small facility tucked between stores in a south Las Vegas strip mall, Switch has quietly built and filled six data centers in Las Vegas with a list of marquee customers.

The SuperNAP marks the coming-out party for Switch, which will now use the enormous footprint to offer its high-density solutions to a broader world of clients seeking to solve difficult power and cooling problems.

The SuperNAP takes Switch’s data center design concepts to the next level. Roy believes the facility will set a new standard for data center management, and expects that its innovations will ultimately be widely adopted by competitors.

“We expect that our competitors will try to replicate our design,” says Roy. “But they can’t build one like it for another two years. So we have two to three years, and by then we will have 1 million square feet. We have the ability to build three more of these on this site, each at 400,000 square feet.”

Does the SuperNAP represent the future of high-density computing? Or is it a unique opportunity enabled by Switch’s unusual bandwidth access and the Las Vegas climate?

Switch which may eventually explore opportunities in other markets, but not before it builds out its full footprint in Las Vegas, where it has advantages in climate and fiber access that aren’t easily duplicated. Roy says bandwidth pricing and telecom relationships are key differentiators for Switch, and that volume pricing has allowed the company to attain significant savings for customers.

In 2002 Switch acquired a former Enron Broadband Services facility in Las Vegas, which had exceptional connectivity due Enron’s efforts to build a commodity bandwidth exchange. Switch says it now has more than 20 backbones running through its bandwidth hub.

“Connectivity wise, there’s not another building in America that comes anywhere near what we can do,” said Roy, who said Switch’s access to fiber backbones is “like pulling up to the Alaska pipeline to get your gasoline.”

“We have direct relationships with (connectivity providers) worldwide,” said Roy. “We’ve really created eight years of amazing relationships with these guys. Those tools are important in the data center and it doesn’t come up that much in our industry. When you go to industry events, all everyone talks about is the infrastructure in your data centers.” Continue on next page …

1 | 2

The dry Vegas climate is also critical to the efficiency of his facilities cooling operations. “You can really efficiently cool very dry air,” said Roy. “It’s hard to create efficient cooling in places where it truly human. And blending hot and cold air is ridiculous.”

So is Switch a unique, location-specific opportunity? Or can others apply parts of these approaches to improve high density data center design? Roy says Switch has filed 26 patents covering the innovations in its Las Vegas operations.

The company’s T-SCIF design builds upon several existing approaches, combining a slab floor and overhead cooling (a design option seen at Equinix facilities) with complete hot-air containment and a ceiling plenum for hot air return (similar in concept to Oracle’s design of its Austin data center). Switch’s design combines the best of both those approaches, and adds its own refinements. In NAP4, the ceiling plenum returns air outside the data center into a “heat aisle” between server rooms. The CRAC units are inserted into openings in the wall, but turned backwards so they draw air from the heat aisle, cool it and return it into the server area.

“We contain all the heat from day one,” said Roy. “Heat is 100 percent contained and processed into heat ceilings. Cold air is dropped past all 42U servers. We’ve tested and proven this by thermal imaging. The size on the (cold air) duct work can support 24kW of CFM for each rack. It’s completely modular and can be adjusted to suit lower density requirements.”

The innovations at the SuperNAP will be harder to replicate. The entire facility is designed around an air-conditioning system that places the custom WDMD cooling units outside the facility. Roy says that incorporating large openings for cooling required reinforced concrete, making it unlikely to recreate outside of new construction. The building is 1,100 feet long and more than 300 feet wide, with a 30-foot ceiling height that allows for a 15-foot heat ceiling plenum.

A tour of Switch’s data centers shows the evolution of its approach from a traditional design to its latest innovations. As customers sought higher densities, Roy and his team spent a year attending conferences and consulting with industry experts, and were frustrated by the process.

“The designs were based on 10 or 20-year-old patents, and they’re still built today like they were 10 years ago,” said Roy. “We said ‘we have to stop looking to the industry for a solution. We have to build it ourselves.’ We got outside that box, and talk to creative people addressing problems similar to yours.”

In Vegas, that meant the casino industry. “The casinos are extremely unique,” said Roy. “They’re glass buildings in the desert. They put large amounts of research and development and to cooling systems. They all did it a little differently.” Roy said those discussions led him to combine a number of the casinos’ techniques into the design of the WDMD cooling units.

As it implemented its T-SCIF design, Switch began winning deals with major technology companies. Roy is not bashful in asserting the merits of Switch’s infrastructure and design innovations. Some prospects were initially puzzled by the claims they heard from Roy and his team, but soon became believers.

“We were consolidating some of our data center operations, and were fascinated with their proposal,” said Dan Butzer, Sun Microsystems Network.com operation. “We assumed going in that because we would need to cool 20kW racks, we’d wind up with water or some kind of location solution.

“We’ve been very impressed with the technology,” Butzer said of Switch. “They promise a lot, and they deliver a lot. These guys know their stuff. We’re planning to significantly expand our footprint when the SuperNAP opens. They keep their word, and they do what they say they’re going to do. It takes a lot of complexity off my plate.”

Making a believer out of Sun helped Switch win other clients, including some of the Internet’s largest companies. Cisco Systems has also acknowledged being a Switch customer. Roy can’t speak publicly about most of his other clients, but they are companies that are serious about high-density computing.

“We haven’t had a client below 400 watts per square foot for a year,” said Roy.

1 | 2

For more news about the SuperNAP, visit our Switch Communications channel. For additional video, check out our DCK video archive and the Data Center Videos channel on YouTube.

The key to Switch’s high-density cooling is a design known as Thermal Separate Compartment in Facility (TSCIF), according to company co-founder Rob Roy. The ingredients in this approach include high-capacity AC units placed outside the data center area, and a tightly integrated hot aisle containment system for the racks. Here’s an overview:

• The cabinets are set on a slab, with no raised floor.
• Chilled air is delivered into the cold aisle near the ceiling rather than through the floor, and enters the cabinets through the front.
• Each cabinet fits into a slot in the TSCIF unit, which encapsulates the rear and sides of each cabinet, while the open front extends beyond the enclosure.
• The hot aisle containment system delivers waste heat back into the ceiling plenum, where it can be returned to the chiller.

Some photos of the TSCIF system can be seen here, and more images and diagrams are available on the Switch web site. A number of data center providers forego a raised floor for overhead cooling, most notably Equinix (EQIX). Heat containment systems are also becoming more widely used.

Switch says the combination of those techniques, along with custom cooling equipment, enables it to handle unusually high power and heat loads. Roy says the data center cold aisle is maintained at 68 degrees, while the temperature in the hot aisle reaches well above 100 degrees, creating a heat differential of nearly 40 degrees.

The key benchmark in designing the system, according to Roy, is the cubic feet per minute (CFM) of cooling that can be pushed into the equipment area. “It’s the most important piece to do high density,” said Roy, who said he focused on CFM performance in designing the air conditioning systems for Switch. That TSCIF system will also be used in the company’s new SuperNAP, a 407,0000 square foot data center that will be supported by 30,000 tons of redundant (system-plus-system) cooling and 30 cooling towers, with a capacity of 4.5 million CFM.

The cooling benchmarks for the Switch facilities are bound to receive scrutiny as the company prepares to open the SuperNAP. There are already some in the industry who are skeptical of Roy’s claims, according to coverage by Ashlee Vance at the Register.

The Register also noted that some high profile technology executives have vouched for Switch’s performance, including Sun Microsystems Chairman Scott McNealy and executives at Cisco Systems (CSCO). “In my opinion Switch has the finest data centers available anywhere,” David Matanane, the senior manager of hosted services at Cisco Systems, told The Register. We spoke with one industry source, who is familiar with the Las Vegas data center scene. “Rob Roy is the real deal,” he told us.

One thing is for certain: at a time when many data center operators are struggling to cool high-density server installations in their facilities, the technologies that Switch Communications is rolling out for its Las Vegas SuperNAP will prompt additional discussion about new approaches to cooling.

Rob Roy, co-founder of Switch Communications, says his company is the best-kept secret in the data center industry. After operating for eight years in stealth mode, serving a client base of military government and military and government customers and large Internet companies, the SuperNAP represents a coming out party for Switch and the data center technologies it has developed.

After years of media silence, Roy has begun discussing Switch’s operations and ambitions for the SuperNAP. The first mention of the SuperNAP leaked out May 5 on the blog for Silverback Migration Solutions, a Switch customer. This past weekend Roy was profiled in The Register, which also received a tour of one of Switch Communications’ five existing Las Vegas data centers. There’s also now a company website, providing information about Switch’s operations and a video preview of the SuperNAP.

We’ve spoken recently with Roy, who was enthusiastic about the data center technologies developed by Switch, and dismissive of those in the industry who might view his claims as improbable. “This is an industry of naysayers,” Roy said.

Roy says Switch operates a room in one of its data centers with cabinets for a name-brand customer running at 1,462 watts per square foot. Roy predicts that the SuperNAP will also be able to support customer power loads of 1,500 watts per square foot.

That’s triple the 450 to 500 watts a square foot that many in the industry view as the upper limit for air cooling. It even exceeds the power loads that Microsoft says it is running inside its CBlox high-density data center containers for cloud computing.

Switch Communications has been operating since 2000. Its business got a huge boost in December 2002, when it acquired a former Enron broadband services facility out of bankruptcy (court records show the 12,000 square foot facility was purchased by Colo Gateways for $930,000). Enron had been seeking to build a commodity bandwidth exchange, and had arranged exceptional connectivity for its Las Vegas center. Switch says it now has more than 20 backbones running through its bandwidth hub.

That connectivity will be one of the primary selling points of the SuperNAP, a 1,100 foot long facility that is under construction in Las Vegas. The data center will be supported by a 250 MVA power substation, and Switch says it will have 146 MVA of generator capacity, suggesting that the building may eventually have as many as 70 two-megawatt generators.

Roy says the SuperNAP will hold 7,000 cabinets once it is fully built out. The facility will be developed in phases, and Roy says the company already has demand for up to 2,000 cabinets from existing customers. A grand opening is scheduled for November.