Tsunamis and Virtual Disaster Recovery

This morning’s news was filled with dramatic footage of earthquake and tsunami damage in Samoa, reminding us anew of the destructive power of tsunamis. While the loss of life and property damage is the primary concern in these events, these type of disasters also prompt reflection for those of us in the data center business. The question: is my company and/or data center prepared for something similar?

A good data center site location process will be aware of flood risk. But for facilities that need to be in coastal areas, do tsunamis represent an unanticipated risk? One industry professional recently used a hypothetical tsunami to illustrate potential future approaches to disaster recovery. Greg Ness at Infrastructure 2.0 describes a recent presentation by WMware’s Mark Thiele, who described how to use virtualization and the mobility of virtual machines to allow rapid redeployment of assets when disaster approaches.

Insurer Plans Data Center in Lansing, Mich.

Jackson National Life Insurance Co. is planning to build a $26 million data center at a site in Lansing, Mich., reports the Lansing State Journal. The new facility is designed for expansion and can “serve as a backup in case anything happens” to the company’s primary data center at its headquarters, which is also in Lansing. The $21 million primary data center was completed in Jan. 2008.

Jackson National said it hasn’t finalized a decision. The company is looking to local officials for $1.4 million in brownfield redevelopment funds for its proposed expansion into a 31,700 square foot former warehouse at 2005 Seager Street. Lansing officials want to keep the high-paying IT jobs in town, and are keen to redevelop the warehouse site, which was previously occupied by a trucking company.

The project would undoubtedly be good for Lansing, but raises a familiar question: how much distance is needed between a primary data center and a backup disaster recovery site? It looks like the proposed expansion site is about 12 miles from the company headquarters, where the paper says the primary site is housed.

Roundup: DataPipe, Peak 10, Equinix, APC

Here’s a roundup of some of some of this week’s headlines from the data center and hosting industry:

• DataPipe rolls out 3PAR services. Utility storage provider 3PAR announced Monday that Cloud-Agile partner DataPipe now offers differentiated virtual private array (VPA) and disaster recovery (DR) services as a part of their Stratosphere cloud services, which are powered by 3PAR Utility Storage.  DataPipe’s cloud platform leverages VMware and 3PAR to offer a managed, secure virtualized service. “We now have the opportunity to expand our cloud service offerings in the areas of security and compliance to increase the value we offer our customers,” said Michael Parks, Chief Technology Officer for DataPipe.  The 3PAR ASSURED product allows DataPipe to use remote copy, data replication, DR and automatic data backup to end users.  The 3PAR SECURED product offers isolated, secure virtual private array servies.
• Peak 10 deploys VMware vSphere 4.  Data center operator Peak 10 announced the delivery of Private Cloud Infrastructure Solutions, via VMware’s vSphere 4 platform.  The platform offers customers pools of virtualized resources that federate between on and off-premise environments on-demand and with ease.  By using vSphere 4 Peak 10 customers can maintain the integrity of their data and applications, while migrating to the cloud platform.  Peak 10’s leadership team are in San Francisco this week attending the VMworld 2009 conference.  Enhancements allowed from Peak 10’s vSphere 4 platform include twice as many virtual processors per virtual machine, four times the memory, three-times increase in network throughput, and additional SAN tiers available for high-end databases.

Sungard Builds Grid From DR Workstations

As a disaster recovery specialist, Sungard Availability Services has an armada of workstations in its facilities for that “what if” moment, ready to provide an instant operations center in the event a disaster affects a client. But what about the rest of the time?

The company has decided to put these computers to work through a new initiative called the Responsible Computing Project. Sungard is using more than 1,000 disaster recovery workstations in its Philadelphia, New York and Chicago data centers to create a computing grid that will perform processing for scientific and humanitarian projects.

When workstations are needed for customer testing or disaster recovery, the Responsible Computing Project allows volunteer jobs to be immediately suspended. The workstation is then automatically converted to its customer’s desktop image, with no impact to either the volunteer project or the customers’ recovery objectives. When the workstations are no longer in use, the volunteer computing projects will resume automatically.

“The Responsible Computing Project exemplifies SunGard’s commitment to harnessing our available resources to address scientific problems, aid in humanitarian efforts and benefit the greater good,” said Steve Buckus, vice president, operations, recovery services at SunGard Availability Services.

The Data Bunker Boomlet

The futuristic Bahnhof data center, located 100 feet beneath Stockholm, is one of many facilities built in nuke-proof subterranean bunkers.

The expansion by The Bunker reflects the growing niche for underground “nuke-proof” data storage facilities housed in former military facilities, mines or limestone caves. These subterranean fortresses have strong appeal for tenants seeking ultra-secure hosting that will survive any eventuality – including a nuclear blast.

This trend has given new life to aging military bunkers in the US, UK and Canada. Although security is usually the primary motivation for customers, underground facilities offer advantages to the data center operator. Chief among them is cooling, as these subterranean facilities typically have a natural temperature of 60 degrees or lower.         

There are challenges as well, especially managing humidity that can be harmful to servers. Underground data centers also can’t receive certification under the LEED (Leadership in Energy and Environmental Design) program for energy efficient buildings because the U.S. Green Buildings Council’s standards has no provision for subterranean facilities.

Here’s a list of all the underground data bunker projects we’ve been tracking at Data Center Knowledge:

• Bahnhof Pionen: Better known as the “James Bond Villain” data center, this former military bunker is 100 feet underneath Stockholm, Sweden and features waterfalls, greenhouse-style NOC, glass-enclosed conference room “floating” above the colocation floor, and blue-lit diesel engines.
• SmartBunker is an ultra-secure data center located in a former NATO command bunker in Lincolnshire, UK. The power used within the 30,000 square feet data centre is generated entirely from wind energy. SmartBunker says it is the first UK facility with no carbon emissions.
• The U.S. Secure Hosting Center is an underground colocation center in Iowa that hosts the web infrastrucutre for Wikia, one of the projects of the Wikimedia Foundation.    
• Iron Mountain hosts data center operations for Marriott Corp. and other customers in its huge data storage facility located 220 feet underground in a limestone cave outside Pittsburgh. The 145-acre facility has its own fire company, water treatment plant and 24-hour security and maintenance force.
• Montgomery Westland, previously known as the Westlin Bunker, operates 40,000 square feet of underground data center and office space in Montgomery, Texas. The facility was initially built by Ling-Chieh Kung, a nephew of Chiang Kai-shek and founder of Westlin Oil. Fearing a nuclear war, the reclusive Kung built a nuke-proof survival shelter.
• The Mountain Complex is a disaster recovery data center in a former mine built into the side of a dolomite mountain in the Ozarks near Branson, Missouri. The facility houses backup data for thousands of financial institutions.

Building a Bigger, Greener Bunker

The entrance to an ultra-secure data center operated by The Bunker in Kent, UK.

The Bunker in Kent, UK has been one of the success stories in the niche market for ultra-secure “military grade” data centers housed in nuke-proof subterranean fortresses. Many data bunkers have an abundance of security features, but The Bunker also has an abundance of customers.

The Bunker, which has hosted clients in underground facilities since 1994, is now expanding with a 12,000 square meter (130,000 square foot) data center on its existing campus in Sandwich, Kent.

The Bunker Secure Hosting Ltd. received final planning permission this week for The Bunker 2, which will offer hosting and disaster recovery services from the N+1 facility. As with the company’s two existing data centers, The Bunker 2 offers space for companies that need to be in the London market but are wary of the risk of disaster or terrorism. The design for the new facility will focus on energy efficiency, typically a secondary concern for ultra-secure data storage facilities.

The new data center “will provide a highly secure, low risk Internet hub outside of London to service clients who wish to house internet-connected systems in the UK, without being reliant on high risk London zones,” said Peregrine Newton, CEO of The Bunker.

Colo5 Buys Jacksonville Facility

Less than a year after opening its doors, Colo5 has acquired its second data center in Jacksonville, Florida, the company said Sunday. Colo5 said it had purchased a 20,000 square foot free-standing data center in Jacksonville to expand its colocation and data recovery services business. Terms of the purchase were not disclosed.

Colo5 operates a disaster recovery center in a 122,000 square foot former Barnett Bank facility on Spring Park Road in Jacksonville, which came online in September. The facility includes a 30,000 square foot data center and 50 disaster recovery suites. The company said it expects Colo5 expects to complete the buildout of its new facility and begin welcoming new customers within the next three months.

More on Carbonite’s Data Loss

Carbonite CEO Dave Friend e-mailed us with additional information about the company’s recent lawsuit against a vendor, in which the company disclosed that it had lost data belonging to 7,500 customers. Friend didn’t deny that the lawsuit states that Carbonite “lost the backups of over 7,500 customers, but says number of customers who actually lost data – rather than having a snapshot of their data disappear – was much smaller. In the interest of thoroughness, here is Friend’s account of the incident:

The data loss event discussed in the lawsuit happened over a year ago. We do not say this to minimize the matter, but it’s important for your readers to know that we stopped buying the servers that caused the problem a long time ago. This is
not a current problem.

The total number of Carbonite customers who were unable to retrieve their data was 54, not 7,500. Here is what happened: The Promise servers that we were purchasing in 2006 and 2007 use RAID technology to spread data redundantly across 15 disk drives so that if any one disk drive fails, you don’t lose any data. The RAID software that makes all this work is embedded as “firmware” in the storage servers. In this case, we believe that the firmware on the servers had bugs that caused the servers to crash. Carbonite automatically restarted all 7,500 backups and more than 99% of these were
completely restored without incident.

Can Amazon be A Player in Disaster Recovery?

Amazon Web Services today introduced a new pricing model for its EC2 compute-on-demand service, offering customers the ability to reserve large amounts of capacity for future use. The feature, known as Reserved Instances, is the latest in a series of steps by Amazon to make AWS more attractive to enterprise users. In introducing the new pricing, Amazon cloud evangelist Jeff Barr highlighted a particular use case.

“Quite a few customers actually told us … they were interested in using EC2 but needed to make sure that we would have a substantial number of instances available to them at any time in order for them to use EC2 in a DR (Disaster Recovery) scenario,” Jeff writes. “In a scenario like this, you can’t simply hope that your facility has sufficient capacity to accommodate your spot needs; you need to secure a firm resource commitment ahead of time.”

This is an interesting pitch, with implications for Amazon’s data center infrastructure. Here’s why: