Water Main Break Floods Dallas Data Center
IT systems in Dallas County were offline for more than three days last week after a water main break flooded the basement of the Dallas County Records Building, which houses the UPS systems and other electrical equipment supporting the data center on the fifth floor of the building. The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one.
The 90-year-old water main ruptured at about 10:30 p.m. last Monday, flooding the basement-level mechanical room with six feet of water, forcing the shutdown of the data center. Power was not restored to the facility until early Friday, as it took several days for the county to pump out the basement. The water was mixed with oil and mud, and needed to be pumped into taker trucks rather than the sewer system, delaying the recovery.
“The county’s criminal justice system nearly ground to a halt, as paper processing from another era led to lengthy delays,” the Dallas News reported. Several courts have gone paperless in recent months, and the district clerk’s office has been converting files to electronic form and destroying paper versions, the paper said.
The county had been warned in a 2008 consultant’s report that it needed to address its lack of a backup data center, but never followed through on the recommendation. The county IT staff is now developing a failover plan and expects to submit an updated plan in two weeks.
A portable generator and UPS were used to restore power to the facility early Friday, and the county IT staff began checking to ensure servers were properly restarted. The Texas county said it believes no critical data was lost.
10:30 p.m. Monday, How? Its only 5:50PM Monday here in the East coast, Its only 4:50 there now.
[...] writes “IT systems in Dallas County were offline for three days last week after a water main break flooded the basement of the Dallas County Records Building, which houses the UPS systems and other electrical equipment [...]
MattPosted June 7th, 2010
I know of at least five states in the US with Dallas counties. Does this story refer to the one in Texas?
uhPosted June 7th, 2010
“I know of at least five states in the US with Dallas counties. Does this story refer to the one in Texas?”
MelPosted June 7th, 2010
SimonTek – … offline for three days “last week”.
So, 10:30pm last Monday night. That’s how.
Matt – Yes, the one in Texas
Captain ObviousPosted June 7th, 2010
Yeah you might want to read all the way to the end there.
“The Texas county said it believes no critical data was lost.”
Yes, it’s the one in Texas.
Larry Jansch, CBCPPosted June 8th, 2010
Their servers survived a soaking in six feet of water, mud, and oil with no data loss? Where can I order fifty of these for our data center?
In a recession, business continuity is one of the first “extravagances” to be cut and one of the last to be restarted, despite insurance carrier’s and legislative requirements of continuous BC coverage.
Larry: The flood affected the power room. The servers were on an upper floor.
Tuesday data center tidbits:bailing out the data center basement, the great data center tax giveaway, and more! « The Server RoomPosted June 8th, 2010
[...] } First up today is the piece about the flood in the Dallas County data center, caused by a ruptured water main. Not having backup servers is bad enough, but putting your backup power equipment below grade is [...]
[...] hacker who hacked into NASA to search for UFO secrets, Chinese boot camp for computer addiction, Water main in a data center breaks, flooding the building, backing up using an FTP service and more! — I backup using Interact [...]
[...] 13, 2010 Haggai, our CTO, stumbled upon the story of a 90-year old water main in Dallas (pre-PC era! imagine that) that burst and flooded a basement, [...]