You may need to scope out cold-weather locations for your next data center build if current climate trends continue. This is what Twitter found after a whistleblower revealed its Sacramento data center was pushed offline due to California’s extreme heatwave.
Whistleblower and former head of security for Twitter Peiter “Mudge” Zatko told CNN and the Washington Post the following about the incident, “Even a temporary but overlapping outage of a small number of datacenters would likely result in the service [Twitter] going offline for weeks, months, or permanently.”
Twitter admitted to the downtime incident via an internal message from the firm’s vice president of engineering Carrie Fernandez, according to the CNN report. In a message to engineers Fernandez said, “On September 5th, Twitter experienced the loss of its Sacramento (SMF) data center region due to extreme weather. The unprecedented event resulted in the total shutdown of physical equipment in SMF.”
Such a disclosure, while breathtaking, isn’t uncommon. This year alone two hyperscale data centers were knocked offline due to summer heat waves more intense than any in recent memory. During the week of July 19th both Google and Oracle’s data centers succumbed to London temperatures upwards of 104 degrees Fahrenheit.
According to Zatko’s testimony to the U.S. Senate Judiciary Committee yesterday, Twitter’s data centers could have withstood Sacramento’s heat streak if only they prioritized engineers’ feedback over large compensation packages.
“To put it bluntly, Twitter leadership ignored its engineers because key parts of leadership lacked the competency to understand the scope of the problem,” Zatko said in his opening statements to the committee. “But more importantly, their executive incentives led them to prioritize profits over security.”
Because of “insufficient data center redundancy” Twitter wouldn’t be able to withstand even a slight outage, revealed Zatko in his initial whistleblower statements.
It’s evident security and uptime go hand-in-hand for data center operations. And while Zatko’s statements hold truths many already embrace in the data center world, lessons can still be gleaned from Twitter’s data center ‘meltdown.’
One may think moving critical workloads to the cloud is the solution. For some, it’s simply delaying disaster or even shifting responsibility to another entity which may or may not have the resiliency needed to run a successful data center operation.
