Here's a cautionary tale about data center fire protection. A small data center in Green Bay, Wisconsin was wiped out by a fire earlier this month, leaving a number of local business web sites offline. The March 19 fire destroyed 75 servers, routers and switches in the data center at Camera Corner/Connecting Point, a Green Bay business offering IT services and web site hosting.
Local media coverage has focused on the recovery effort. CEO Rick Chernick has emphasized the fact that the company had fire insurance, and said that alarms helped mitigate the potential damage. "If the equipment we had in place to warn us didn't register fast enough, notifying the fire department, getting our people down here, we wouldn't have a business today probably," Chernick said.
But it took 10 days to get customer web sites back online, indicating the company had no live backup plan. That was a problem for one customer, the Green Bay Blizzard arena football team, which was counting on selling tickets online for yesterday's home opener.
While the company discussed the usefulness of its fire alarms, it didn't address whether the data center had a fire suppression system. But it doesn't sound like it. The Green Bay Press Gazette describes "racks of blackened, melted plastic and steel." We've previously looked at data center fire suppression tools and how they have evolved with the industry's recent focus on environmental considerations.
Camera Corner recently decided to take data center space in Milwaukee, but Chernick said that wouldn't be ready for another 60 days. The CEO insisted the company had thought a lot about disaster recovery:
"As part of your plan: Where are you going to go? Where are you going to set up? Where are you going to put phones in?" Chernick said. "We had all that planned out so we knew where we were going to go ... if we had a fire."
And yet customer web sites were offline for at least 10 days. The local media didn't ask tougher questions about whether that was adequate.
Sometimes customers don't ask enough questions, either. I realize this is a small local data center. But it provides a lesson for companies shopping for web hosting services: ask a lot of specific questions. There's always some kind of "disaster recovery" plan. How about: "If a disaster hits your data center, how long will my web site be offline?" If the answer is couched in days or weeks, find another provider. If local companies can't meet your requirements, there are plenty of regional and national companies that can.
Here are some pointed questions to ask potential providers:
- What kind of data backup systems are in place?
- Do you have more than one data center? Some SaaS providers operate without a backup facility, so make no assumptions about this.
- How often does a provider test its backup systems? Ask specifically about diesel generators, which are often a problem when not exercised monthly.
And ask them to describe their data center fire suppression systems. Many data center operators prefer gas-based fire prevention systems to sprinkler systems using water, which could damage equipment in the event of a leak. Others use water in a "pre-action" system in which water fills the sprinkler pipes only upon an alarm. If their fire suppression system is that the fire department is just down the street, that's not a good sign.
