Diesel ‘Bucket Brigade’ Keeps Peer 1 Online at 75 Broad
October 31st, 2012 By: Rich Miller
A determined team of employees of Peer 1 Hosting, blog host Squarespace and Fog Creek Software have formed a “bucket brigade” to relay 5-gallon buckets of diesel fuel up 17 flights of stairs at 75 Broad Street to refuel a generator providing emergency power to the Peer 1 data center.
It is a decidedly low-tech, brute-force solution to the challenges at 75 Broad, the hobbled data center building in Lower Manhattan. Strong arms and strong backs are replacing fuel pumps that normally bring diesel from tanks in the basement to the upper floors of the building, where the backup generators are providing emergency power to keep the servers online in the Peer 1 data center. Those pumps shut down yesterday when the storm surge from Hurricane Sandy flooded the basement levels of 75 Broad, rendering the pumps unusable.
Peer 1 initially planned for a “controlled shutdown” at 10:45 a.m. While the generators chugged on past that point, the company and its customers developed an alternate plan -
“”We have 25 people stationed throughout the building to relay more fuel to the roof once it arrives,” Squarespace reported on its status blog. “The building now has powerful pumps clearing out the basement, which we hope will expose the main pump lines — which would allow us fuel for days. Hopefully our manual efforts, joined with the building, can see us through. … ”Spirits are strong and everyone from Peer1, Fog Creek, and Squarespace is working together.”
Squarespace acknowledges the effort is “not sustainable” over the long haul. Both Peer 1 and Internap are working on long-term solutions should the pumping and repairs require extended time. But the bucket brigade reflects an unusual physical embodiment of the depth of commitment to keep data centers online, come what may.
UPDATE: As of 1 p.m. Eastern the generator’s day tankw as full, allowing the bucket brigade to take a 90-minute break. But little progress was being made on long-term resolutions. “The building’s first attempt at an alternative method for pumping fuel to the 18th floor has failed, as the fuel pump wasn’t powerful enough,” Squarespace reports. “They believe they have sourced an alternate pump, but given the situation in New York City right now, we’re in a wait-and-see posture. Fuel- and water-pumps are in short supply.”
Perhaps more problematic is the stats report on the flooding. “The basement is not draining at all either, despite the large pumps that were brought in late last night. DEP and ConEd have been here for a few hours. They fear a water main has ruptured somewhere and is pushing water (and other stuff) into our basement as we pump it out. This is pure speculation at this point.”
JohnPosted October 31st, 2012
neuromancerPosted October 31st, 2012
Why not rig up a block and tackle out of a window and haul up jerry cans much safer than carying open buckets of fuel.
kirstenPosted October 31st, 2012
thanks guys! you rock I am using Trello to work on some serious deadlines and reaalllllly appreciate the hard work to keep it up and running.
KenPosted October 31st, 2012
Probably takes longer and more chances of spilling. They do make covers for 5 gallon pails you know…
fredPosted October 31st, 2012
Couldn’t someone have gone to a Walmart or somethng and bought proper gas containers?
fredPosted October 31st, 2012
I wonder what a fire marshall thinks of open pails of diesel being carried up 17 flights spilling all over the place and spreading diesel fumes.
AdamPosted October 31st, 2012
They may not have the ability to get to a OPEN Walmart to go buy something that I’m sure a lot of other people also bought. I’m sure the fire marshal is busy with other things right now given the state of NY.
There actually aren’t any Wal-Marts in NYC. But the greater point is that there isn’t much of anything available in the city at the moment.
Kirsten, Trello has been moved to AWS. It’s simple to deploy and has a relatively small amount of data and through a different type of hard work they were able to move it. I wish Trello was being powered by a bucket brigade because I use it and that would be awesome, but alas, Trello is safe from the situation in NYC.
Chuck O'DonnellPosted November 1st, 2012
As far as I am concerned, and I think I speak for a fair number of people up whose livelihoods depend on this data center staying up, these guys are heroes.
Carry on gentlemen, carry on.
[...] Incredible story was reported by Barb Darrow from GigaOM and Rich Miller from DCK. During the hurricane, several Manhattan (NYC) streets and buildings were left flooded. One of [...]
@fred, diesel has a really high flash point (50C +) where Gasoline has a flashpoint of -65C (that is over 100C difference). Unless it is over 50C in the building there should be very few ‘fumes’ coming from the diesel. The high flash point is kind of the reason it contains so much more energy than the same mass of gasoline, and also why it is a superior fuel to gasoline in every way EXCEPT cold starting, gasoline blows diesel out of the water to start below freezing.
Probably not the best idea to carry it in open containers.
MikePosted November 2nd, 2012
Would have been cool if the Peer1 a$$ hats would have not lied to other people in the building. They told other people that they were filling the common generator header to get them to pitch in and help. Turns out they were only filling their own… Way to go… great company!
ScottPosted November 4th, 2012
No open containers were used to carry diesel. It would be very difficult to fill a closed bucket. Try finding out what really happened before posting. Every container was closed securely. Mike, you are extremely clueless Everybody involved was aware that it was done to keep Peer1′s generator running.