GitHub, the most popular online repository for open source code and hosting services, went down for two hours Thursday due to a power outage in its primary data center.
“A brief power disruption at our primary data center caused a cascading failure that impacted several services critical to GitHub.com's operation,” Sam Lambert, GitHub’s director of systems, wrote in a status update on the company’s blog Friday morning. “While we worked to recover service, GitHub.com was unavailable for two hours and six minutes.”
The outage started around 4pm Pacific. The final procedure to fully restore the facility’s power infrastructure was completed Thursday evening.
Utility power outages do not bring down data centers in most cases, since these facilities are designed with UPS units, backup generators, and transfer systems that fail over to the generators automatically. When they do happen, power-related data center outages are caused by failure of those backup systems.
A recent study by the Ponemon Institute, sponsored by Emerson Network Power, found that UPS failure is the most common cause of data center outages, followed by cybercrime (namely DDoS attacks), human error, and cooling system failure, in that order.
It’s unclear where San Francisco-based GitHub’s primary data center is. Lambert told us in an earlier interview that the company does not disclose its data center locations.
GitHub’s Bare Metal Cloud
What we do know is that it has to expand its data center capacity rapidly to keep up with growing popularity of open source software. “We have a massive intake of new repositories and just new data, so we’re continually expanding our storage infrastructure,” Lambert said.
GitHub doesn’t use virtual machines. Instead, it has built a bare-metal cloud, which allows it to provision physical machines the same way cloud VMs are provisioned.
“We deploy onto physical machines,” he said. “We have a system internally that allows us to deploy physical machines as the cloud.”
Effects of the outage were widely felt across startup and enterprise developers around the world, many of whom use GitHub for coding day-to-day. Here is a collection of Tweets from Thursday that illustrate the carnage:
— Zhuowei Zhang (@zhuowei) January 28, 2016
Devs when GitHub is down: "plastic lightsaber fight!" Devs when StackOverflow is down: "everyone's going to find out I can't actually code"
— SecuriTay (@SwiftOnSecurity) January 28, 2016
Github is down, so it's the San Francisco equivalent of a snow day for the majority of the city. — Emily Gilbert (@Emily_A_Gilbert) January 28, 2016
@Emily_A_Gilbert Gitzzard. Gitmageddon.
— Tarak Parekh (@tparekh) January 28, 2016