South Korean tech giants Kakao and Naver were the latest companies hit by a data center fire. Around 3:30 p.m. local time last October 15, a fire broke out at the SK C&C building in Pangyo, just south of Seoul, which houses the data centers for the two firms. The blaze caused login issues and messenger delivery failures for Kakao, while some of Naver’s services were down, including search, news, and shopping.
Kakao said that a “lithium-ion battery in the building’s electrical control room caught fire and caused the outage,” but “police and fire authorities are still investigating the main cause” and it is still “unclear whether the fire was a result of a battery defect,” according to reporting from The Korea Times.
Consumers and Government Demand Action
The massive outage from the data center fire had wide-ranging effects for the millions of users — including companies and small businesses — who depend on Kakao’s services, many of whom are contemplating filing a class-action lawsuit against the tech firm to claim damages suffered during the outage. Kakao plans to compensate users and has formed a committee to establish a compensation policy.
Meanwhile, in response to increased pressure for immediate action, South Korean officials raided SK C&C’s headquarters to search for the true reason behind the data center fire, prompting increased government scrutiny and more regulations ahead for the country’s data centers.
Three Data Center Fires in Only Two Years: Why?
The data center fire in South Korea is the third such incident in two years. In August, one of Google’s data centers caught fire due to an electrical explosion (specifically an arc flash), injuring three electricians working on a substation near the data center. In 2021, a fire completely destroyed one of OVH’s data centers in France and partly damaged another, though no injuries were reported.
The risk of a data center fire occurring is rare according to the Uptime Institute. Of the 8,000 abnormal incidents shared by the institute’s members since 1994, only 11 fires in data centers — or less than 0.5 each year — were recorded. From 2016 to 2020, the institute’s database held only 13 records of publicly reported outages caused by fire.
Data center fires may be an infrequent occurrence, but when they do happen, their consequences are typically devastating. In the South Korea incident, for instance, Kakao could lose billions of dollars in revenue and face increased costs.
Best Practices to Protect Against Data Center Fire Risks
To reduce the risk of fires occurring in data centers, the Uptime Institute offers the following advice:
- Detection: Use VESDA (very early smoke detection apparatus) systems.
- Suppression: Ensure appropriate fire barriers and separation of systems are in place, and keep fire suppression systems — be it water-based, gas-based, or a combination of both — well maintained.
- Mitigation: Conduct periodic risk assessments to identify and address any issues with these systems.
“... data centers must consider fire prevention and mitigation systems that go far beyond minimum local code requirements," Ryan Orr, vice president, Topology Services, Uptime Institute told Data Center Knowledge.
"While the sector has made significant progress in the use of low flammability materials and reducing fire risk by careful material selection, data centers continue to rely on equipment and components that are capable of both starting and increasing the severity of a fire; and the impacts should a fire occur need to be seriously considered at both the design stage, and during ongoing operations.”
While detection is a critical first step in protecting against fires, mitigation and suppression are the next crucial elements. Some of the contributing factors to the OVH fire, for example, were the lack of an automatic fire extinguisher system, “a wooden ceiling rated to resist fire for only an hour, and two inner courtyards that acted as ‘fire chimneys,’” according to reporting from The Register.
Having a comprehensive and consistently updated disaster management protocol is also imperative. In the case of Kakao, the company did have a disaster management protocol in place, but that protocol did not account for the power outage at the time of the fire, slowing down service restoration efforts. Learning from this incident, Kakao has put together a recurrence prevention committee that will outline measures to prevent a similar event from happening.
In addition to putting all the necessary detection, suppression, and mitigation measures in place, data center operators must also develop a robust disaster recovery plan, which includes backup, replication, and recovery procedures, as well as processes to ensure rapid response and swift restoration of services and operations.
While Naver’s services went down, they were restored within a few hours owing to the company having an off-site backup. On the other hand, Kakao did not have a disaster recovery plan or backup sites, resulting in days-long outages. The company said that once the data center is fully operational, it will build in redundancy within two months to prevent a similar disruption. Kakao is also investing over $300 million to complete its own data center next year, and will start building another data center for completion in 2024
