Is your data center ready for an EF5 tornado (pictured above)? Or a major hurricane? Here's a look at storm preparedness. (Photo by Justin Hobson via Wikimedia Commons)
How do you storm-proof your data center against increasingly fierce tornadoes, hurricanes, derechos and other windstorms? It's a front-of-mind issue for the industry in the wake of a wave of powerful tornadoes and the first named storm of the 2013 hurricane season. On Wednesday and Thursday, violent thunderstorms blew through key data center hubs in suburban Chicago and northern Virginia.
There were no major outages from this week's storms, reinforcing the basic value proposition of the data center. The severity of storms like Superstorm Sandy and the recent EF5 tornadoes in Oklahoma has prompted scrutiny of design standards for mission-critical facilities.
"From a structural standpoint, you can design and construct a facility to withstand just about anything," said Ron Vokoun, Mission Critical Market Leader - Western Region at JE Dunn Construction. " It comes down to how much you want to spend and weigh the extra costs against the benefits of locating in an area with these types of risks.
"If you are located in Florida, is the pull of the ‘server hugger’ mentality so strong that you will spend extra money to build your data center close to home, or will you move it farther inland within the constraints of your latency requirements?" said Vokoun. "Do the incentives being offered by the Midwestern states offset the extra structural costs to build a data center that will withstand an F5 tornado? In many cases they do. It’s all about analyzing risk and TCO to make an informed business decision."
SunGard Offers Early Warning System
Uptime during storms isn't just a function of cement and steel, but also having strong policies and procedures in place. With fast-moving storms like the Oklahoma tornadoes, an early warning can make a huge difference in executing those procedures.
SunGard Availability Services this week announced a new weather alert system leveraging real-time data from NOAA to better inform and prepare customers of weather-related disasters headed their way. The company is leveraging real-time data from the National Oceanic and Atmospheric Administration (NOAA) and using big data analytics to keep its customers abreast of threatening storms and other natural disasters.
"Many companies overlook business continuity during severe storms simply because they aren’t prepared," said Bob DiLossi, director, Crisis Management, SunGard Availability Services. "Through the years, we’ve listened to our customers’ needs and are proud to be among the first IT service providers offering a weather alert system free of charge. By using real time weather information combined with business analytics, we’re preparing our customers for what’s to come, giving them time to plan and take action."
How Sturdy to Build?
Given the EF5 in Oklahoma, questions have been raised about whether a data center could withstood a direct hit. Data centers often claim to meet the “Miami-Dade standard” of being able to withstand 150 MPH wind speeds. Tornadoes can generate winds far beyond these numbers. So is being able to withstand 300 MPH going to be the new standard? Most likely not.
Despite the recent EF5 storms, tornadoes are a far lesser risk than hurricanes or earthquakes. Hurricanes and earthquakes can damage huge swaths of land, whereas tornadoes cause sporadic damage for a few miles, perhaps a mile wide. Ninety five percent of tornadoes are below EF3 intensity, and only 0.1 percent reach EF5 according to NOAA. The bigger threat is outside of the physical data center, in the form of power outages, flooding and network breaks.
Regardless, there are some providers building and marketing facilities as "tornado proof." Perimeter Technology has built its Oklahoma data center to withstand an EF3 tornado. The raised floor portion of the data center is surrounded with 8.5 inch concrete, reinforced walls. It’s surrounded by offices, which are then protected by another 8.5 inch exterior wall. The roof is double reinforced, thick enough to handle a storm’s uplift. It also insulates the building helping with the cooling.
The EF5 on May 20 struck less than 20 miles from Perimeter’s data center.
In 2011, DataCave built a tornado-proof 4.5 million pound roof in Columbus, Indiana. Excessive? Data Cave said it’s based on experience with Midwest tornadoes and the types of damage they can inflict upon structures. Barry Czachura said in our original coverage that the roof is a key line of defense.
Hurricane Season is Here: Batten Down the Hatches
Hurricane season is upon us, having officially starts on June 1. The National Oceanic and Atmospheric Administration (NOAA) Climate Prediction Center anticipates an “active to extremely active” season, according to NOAA’s annual report issued in May. NOAA predicts 13 to 20 named storms this year, including 7 to 11 hurricanes. Three to six of these could be major hurricanes. These ranges are above the typical seasonal average of 12 named storms, six hurricanes and three major hurricanes.
The first tropical storm of the season, Andrea, hit last week and was primarily a rain event.
In preparation for the upcoming season, companies with facilities in Hurricane-sensitive areas are going the extra mile this year. Peak 10, which operates a network of data centers across the southeast, has assembled a response team of IT leaders and engineers from several of its geographic operations to be ready for deployment in the event of a natural disaster. Peak 10 also offers a "Recovery Cloud" option for companies who need the added security of rapid recovery of data lost during a disaster.
“People are the most critical part of any disaster recovery plan,” said Jeff Biggs, executive vice president of operations and technology for Peak 10.” Our national response team ensures that customers have the support they need locally while allowing our employees who are also affected by the disaster to tend to their families and homes. Deploying our response team is an added measure we take to ensure that our customers, and their mission-critical data and IT systems, are taken care of.”
Peak 10 has taken a number of operational steps including standard operating procedure (SOP) reviews for each of its facilities along the East Coast. The SOP reviews ensure appropriate security and supplies are in place, testing redundant network infrastructure and carrier connectivity, and confirming arrangements are in place and vendors are on standby for emergency refueling if an extended utility outage is experienced. The company also regularly tests its emergency power systems throughout the year to ensure they are ready in the event of utility power loss. This includes load testing of Uninterruptable Power Supplies (UPSs) and emergency standby generator systems.
Steps to Readiness
SunGard blogged about National Hurricane Preparedness, telling folks not to roll their eyes at it given what we faced last year. Superstorm Sandy did more than $75 billion in damage. During Hurricane Sandy, SunGard AS received 342 alerts from customers and 117 disaster declarations. To support these declarations, it deployed almost one-third of its staff, sent out 5 mobile recovery units, and filled up 1,500 workgroup seats for their customers’ employees. The Carlstadt, NJ data center served as an impromptu command center for local authorities and took in nearby flood victims.
SunGard recently shared best practices and challenges, based on its experiences during Sandy and the 2,000 other disasters for which the company has provided support since 1990. These include:
- Data protection challenges: "Backing up onto tapes is good, but getting those tapes over to our recovery centers through flooded streets was a challenge," writes SunGard's Maryling Yu.
- Systems recovery challenges: "Not having the right operating systems and servers and storage and networks and hypervisors at the recovery site was a huge challenge for many of our customers. Fully one-third of our customers found themselves having to make serious changes to their recovery site specifications…and by 'serious' I mean, 'We forgot to tell you about a mainframe that we had.'"
- People challenges: "Apparently, telework was not all that great of a DR strategy during Sandy," writes Yu. "This was a regional disaster that saw power outages across a large swath of territory … which meant that work-from-home or work-from-Starbucks was impaired as well."
- Process challenges: "Recovery runbooks were frequently out-of-date. It certainly doesn’t help to have a runbook for recovering Windows 2003 servers when you need to recover Windows 2008 servers," says Yu.
- Program challenges: Change management to sync up production and recovery environments is still not a focus for many of our customers. How can it be? They’ve got enough to do, because they’ve got to observe the ever-louder IT mantra of today: “Do more with less.”
Yu’s full observations can be found here. The company has also offered a Hurricane Preparedness Toolkit (registration required) here.
