Thursday and Friday were all-hands-on-deck kind of days for European staff of the Roubaix, France-based cloud and hosting provider OVH, one of whose data centers in Eastern France was completely destroyed in a fire early Wednesday morning.
OVH teams from other company offices and data centers, as well as HR staff, arrived to support and reinforce the team at the Strasbourg campus, OVH, the largest Europe-based cloud provider, said in a status update posted Thursday. Top priorities locally were to secure and clean up the site and reconnect electricity and network for the three data centers that survived the fire. All maintenance operations in OVH data centers has been "frozen" since Wednesday, presumably to free up resources to help in Strasbourg and to avoid any unexpected problems in other data centers, which often arise during maintenance.
An even higher priority for the company was to set up replacement servers for the affected customers in other OVH data centers. OVH said it had enough new servers stocked at its Roubaix and Gravelines sites for most of those customers. Roubaix and Gravelines are where the company is offering to stand up services for affected customers.
"Two UPS On Fire"
The exact cause of the fire has yet to be determined. The investigation into the cause was still ongoing as of Friday afternoon.
In a video update recorded Thursday, OVH founder Octave Klaba, who'd been onsite since Wednesday morning, said thermal images taken by firefighters who arrived at the scene shortly after the fire started showed two UPS systems (UPS 7 and UPS 8) on fire. One of the UPS's had undergone extensive maintenance procedures a day earlier.
"We had maintenance on UPS 7 in the morning," Klaba said in the video. "The supplier came and changed a lot of pieces inside of UPS7 and restarted UPS7 in the afternoon and it seems that it was working."
When the fire first broke out (just after midnight local time on Wednesday), alarms went off, and multiple OVH staff members immediately came to the area of the alarm, he said. There was so much smoke that they left the building after several minutes. The firefighters arrived shortly thereafter.
Trippling Server Production Capacity
Internal volunteers have reinforced staff at the company’s French server manufacturing site, in Croix. Assembly lines at the factory have been reorganized and tripled production capacity to 3,000 to 4,000 servers per week since Friday morning, OVH said in an update that evening.
Production staff at the factory will work additional weekday hours and during the weekends, the company said. "We are committed to producing nearly 15,000 new servers in the coming weeks," the Friday update read.
Klaba has been posting frequent updates on Twitter, including Thursday's video:
Planning DR for the DR-less
OVH's support teams and technical account managers are working on a "support plan" for customers who hosted applications in Strasbourg and didn't have disaster recovery plans or (for some reason) hadn't activated them. (According to comments on Twitter, the number of affected customers without DR plans was substantial.)
The OVH data center fire knocked 3.6 million websites offline, the UK-based internet security firm Netcraft said Wednesday. The downed sites included those of banks, webmail services, news services, online retailers, “and several countries’ government websites.”
There were no injuries caused by the fire, according to OVH.
Besides the one data center that burned down entirely (SBG2), part of a second one (SBG1) did as well. The entire four-data center campus was de-energized, however, and OVH did not expect to start bringing power back until Monday, March 15, at the earliest.
After some examination, OVH found that all fiber paths that enter the Strasbourg campus were operational.
The SBG2 data center will have to be “almost entirely reconstructed,” OVH said. “SBG1 was heavily damaged.” It found no fire-related issues at SBG4 or SBG3. The company expects to restore the network room at SBG1 early next week.
-- This story, originally posted on March 11, has been updated with more recent, March 12 information
