Water Main Break Floods Dallas Data Center

11 comments

IT systems in Dallas County were offline for more than three days last week after a water main break flooded the basement of the Dallas County Records Building, which houses the UPS systems and other electrical equipment supporting the data center on the fifth floor of the building. The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one.

The 90-year-old water main ruptured at about 10:30 p.m. last Monday, flooding the basement-level mechanical room with six feet of water, forcing the shutdown of the data center. Power was not restored to the facility until early Friday, as it took several days for the county to pump out the basement. The water was mixed with oil and mud, and needed to be pumped into taker trucks rather than the sewer system, delaying the recovery.

“The county’s criminal justice system nearly ground to a halt, as paper processing from another era led to lengthy delays,” the Dallas News reported. Several courts have gone paperless in recent months, and the district clerk’s office has been converting files to electronic form and destroying paper versions, the paper said.

The county had been warned in a 2008 consultant’s report that it needed to address its lack of a backup data center, but never followed through on the recommendation. The county IT staff is now developing a failover plan and expects to submit an updated plan in two weeks.

A portable generator and UPS were used to restore power to the facility early Friday, and the county IT staff began checking to ensure servers were properly restarted. The Texas county said it believes no critical data was lost.

About the Author

Rich Miller is the founder and editor-in-chief of Data Center Knowledge, and has been reporting on the data center sector since 2000. He has tracked the growing impact of high-density computing on the power and cooling of data centers, and the resulting push for improved energy efficiency in these facilities.

Add Your Comments

  • (will not be published)

11 Comments

  1. 10:30 p.m. Monday, How? Its only 5:50PM Monday here in the East coast, Its only 4:50 there now.

  2. Matt

    I know of at least five states in the US with Dallas counties. Does this story refer to the one in Texas?

  3. uh

    "I know of at least five states in the US with Dallas counties. Does this story refer to the one in Texas?" rtfa bro

  4. Mel

    SimonTek - ... offline for three days "last week". So, 10:30pm last Monday night. That's how. Matt - Yes, the one in Texas

  5. Captain Obvious

    Yeah you might want to read all the way to the end there. "The Texas county said it believes no critical data was lost." Yes, it's the one in Texas.

  6. Larry Jansch, CBCP

    Their servers survived a soaking in six feet of water, mud, and oil with no data loss? Where can I order fifty of these for our data center? In a recession, business continuity is one of the first "extravagances" to be cut and one of the last to be restarted, despite insurance carrier's and legislative requirements of continuous BC coverage.

  7. Larry: The flood affected the power room. The servers were on an upper floor.