Vancouver Power Outage KOs Plenty of Fish
An underground circuit fire caused a major power outage in Vancouver yesterday, which disabled traffic lights and cash machines and also led to lengthy outages for web sites hosted at the Harbour Centre telecom hub. The main generator supporting the Harbour Centre failed, cutting power to a data center operated by Peer 1.
Among the sites affected by the five-hour outage were Plenty of Fish, the massively popular free dating site operated by Internet entrepreneur Markus Frind. Plenty of Fish is known for supporting huge web traffic on a lightweight infrastructure, as well as Frind’s ability to generate revenue using Google AdSense. Other sites affected by the outage included the web host services Bravenet and Blue Fur.
Harbour Centre’s main building generator, Gen No.7, started and maintained power as designed, but failed 20 minutes after the loss of commercial power due to a failure in the generator’s cooling system. Gen No.7′s cooling system is dependent upon water provided by the city of Vancouver and requires a certain level of water pressure to function normally. As fire fighters worked to put out the fire that had broken out on Richards Street, water pressure in the downtown core, (including Harbour Center), declined to a point where it caused the generator to overheat and malfunction. Once the generator failed, PEER 1′s UPS systems eventually failed as well once the UPS batteries had drained.
Harbour Centre was able to get the generator working by 3:40 pm local time, and Peer 1 restored power to customers by 4:20 pm. At 5:55 pm the building was back on utility power. Banducci emphasized that Peer 1 would fully honor its customer service level agreements (SLAs) and take steps to avoid a repeat of the outage.
Harbour Centre is a 28-floor skyscraper in the central business district of downtown Vancouver, which also serves as the city’s primary telecom and data center hub. The building’s description of its telecommunications infrastructure states that the property has six diesel generators ranging from 250 kilowatts to 1.5 megawatts in capacity.
I have to wonder why their generators needed some sort of an external cooling system? Beyond being an apparent SPOF, it is simply unnecessary. As a friend said last night “It’s not as if all the cars and trucks on the road drive around with a water hose of infinite length to stay cool”.
JeffPosted July 15th, 2008
@Michael T. Halligan:
Michael, cars don’t need a water hose of infinite length to stay cool because the heat is moved from the coolant to the air as the air flows over the radiator (or by the fan, if you’re not moving). Backup generators that are fixed in place obviously can’t use that method so you have to either find another way to cool the water or replace the hot water with new (cooler) water. That’s why they need a plentiful water supply.
JohnPosted July 16th, 2008
interesting – regarding the 6 generators:
I was there speaking with the maintenance guys and they were refering to the $2 million generator that powers Peer1 and Fusepoint as “Generator 7″ – the one that the cooling failed on….