An underground circuit fire caused a major power outage in Vancouver yesterday, which disabled traffic lights and cash machines and also led to lengthy outages for web sites hosted at the Harbour Centre telecom hub. The main generator supporting the Harbour Centre failed, cutting power to a data center operated by Peer 1.
Among the sites affected by the five-hour outage were Plenty of Fish, the massively popular free dating site operated by Internet entrepreneur Markus Frind. Plenty of Fish is known for supporting huge web traffic on a lightweight infrastructure, as well as Frind's ability to generate revenue using Google AdSense. Other sites affected by the outage included the web host services Bravenet and Blue Fur.
Harbour Centre's main building generator, Gen No.7, started and maintained power as designed, but failed 20 minutes after the loss of commercial power due to a failure in the generator's cooling system. Gen No.7's cooling system is dependent upon water provided by the city of Vancouver and requires a certain level of water pressure to function normally. As fire fighters worked to put out the fire that had broken out on Richards Street, water pressure in the downtown core, (including Harbour Center), declined to a point where it caused the generator to overheat and malfunction. Once the generator failed, PEER 1's UPS systems eventually failed as well once the UPS batteries had drained.
Harbour Centre was able to get the generator working by 3:40 pm local time, and Peer 1 restored power to customers by 4:20 pm. At 5:55 pm the building was back on utility power. Banducci emphasized that Peer 1 would fully honor its customer service level agreements (SLAs) and take steps to avoid a repeat of the outage.
Harbour Centre is a 28-floor skyscraper in the central business district of downtown Vancouver, which also serves as the city's primary telecom and data center hub. The building's description of its telecommunications infrastructure states that the property has six diesel generators ranging from 250 kilowatts to 1.5 megawatts in capacity.