Diesel ‘Bucket Brigade’ Keeps Peer 1 Online at 75 Broad

14 comments

Teams from Squarespace fill buckets with diesel fuel to haul them up 17 stories to the generator keeping the data center online. Staff from Peer 1, Squarespace and Fog Creek Software have formed this unusual Internet bucket brigade. (Photo via Squarespace)

A determined team of employees of Peer 1 Hosting, blog host Squarespace and Fog Creek Software have formed a “bucket brigade” to relay 5-gallon buckets of diesel fuel up 17 flights of stairs at 75 Broad Street to refuel a generator providing emergency power to the Peer 1 data center.

It is a decidedly low-tech, brute-force solution to the challenges at 75 Broad, the hobbled data center building in Lower Manhattan. Strong arms and strong backs are replacing fuel pumps that normally bring diesel from tanks in the basement to the upper floors of the building, where the backup generators are providing emergency power to keep the servers online in the Peer 1 data center. Those pumps shut down yesterday when the storm surge from Hurricane Sandy flooded the basement levels of 75 Broad, rendering the pumps unusable.

Peer 1 initially planned for a “controlled shutdown” at 10:45 a.m. While the generators chugged on past that point, the company and its customers developed an alternate plan -

“”We have 25 people stationed throughout the building to relay more fuel to the roof once it arrives,” Squarespace reported on its status blog. “The building now has powerful pumps clearing out the basement, which we hope will expose the main pump lines — which would allow us fuel for days. Hopefully our manual efforts, joined with the building, can see us through. … “Spirits are strong and everyone from Peer1, Fog Creek, and Squarespace is working together.”

Squarespace acknowledges the effort is “not sustainable” over the long haul. Both Peer 1 and Internap are working on long-term solutions should the pumping and repairs require extended time. But the bucket brigade reflects an unusual physical embodiment of the depth of commitment to keep data centers online, come what may.

UPDATE: As of 1 p.m. Eastern the generator’s day tankw as full, allowing the bucket brigade to take a 90-minute break. But little progress was being made on long-term resolutions. “The building’s first attempt at an alternative method for pumping fuel to the 18th floor has failed, as the fuel pump wasn’t powerful enough,” Squarespace reports. “They believe they have sourced an alternate pump, but given the situation in New York City right now, we’re in a wait-and-see posture. Fuel- and water-pumps are in short supply.”

Perhaps more problematic is the stats report on the flooding. “The basement is not draining at all either, despite the large pumps that were brought in late last night. DEP and ConEd have been here for a few hours. They fear a water main has ruptured somewhere and is pushing water (and other stuff) into our basement as we pump it out. This is pure speculation at this point.”

About the Author

Rich Miller is the founder and editor at large of Data Center Knowledge, and has been reporting on the data center sector since 2000. He has tracked the growing impact of high-density computing on the power and cooling of data centers, and the resulting push for improved energy efficiency in these facilities.

Add Your Comments

  • (will not be published)

14 Comments

  1. John

    Safety 1st! http://www.amazon.com/Blitz-Gallon-Diesel-Approved-31754/dp/B0007M8ZSU

  2. neuromancer

    Why not rig up a block and tackle out of a window and haul up jerry cans much safer than carying open buckets of fuel.

  3. kirsten

    thanks guys! you rock :) I am using Trello to work on some serious deadlines and reaalllllly appreciate the hard work to keep it up and running.

  4. Ken

    Probably takes longer and more chances of spilling. They do make covers for 5 gallon pails you know...

  5. fred

    Couldn't someone have gone to a Walmart or somethng and bought proper gas containers?

  6. fred

    I wonder what a fire marshall thinks of open pails of diesel being carried up 17 flights spilling all over the place and spreading diesel fumes.

  7. Adam

    They may not have the ability to get to a OPEN Walmart to go buy something that I'm sure a lot of other people also bought. I'm sure the fire marshal is busy with other things right now given the state of NY.

  8. There actually aren't any Wal-Marts in NYC. But the greater point is that there isn't much of anything available in the city at the moment.

  9. Kirsten, Trello has been moved to AWS. It's simple to deploy and has a relatively small amount of data and through a different type of hard work they were able to move it. I wish Trello was being powered by a bucket brigade because I use it and that would be awesome, but alas, Trello is safe from the situation in NYC.

  10. Chuck O'Donnell

    As far as I am concerned, and I think I speak for a fair number of people up whose livelihoods depend on this data center staying up, these guys are heroes. Carry on gentlemen, carry on.

  11. @fred, diesel has a really high flash point (50C +) where Gasoline has a flashpoint of -65C (that is over 100C difference). Unless it is over 50C in the building there should be very few 'fumes' coming from the diesel. The high flash point is kind of the reason it contains so much more energy than the same mass of gasoline, and also why it is a superior fuel to gasoline in every way EXCEPT cold starting, gasoline blows diesel out of the water to start below freezing. Probably not the best idea to carry it in open containers.

  12. Mike

    Would have been cool if the Peer1 a$$ hats would have not lied to other people in the building. They told other people that they were filling the common generator header to get them to pitch in and help. Turns out they were only filling their own... Way to go... great company!

  13. Scott

    No open containers were used to carry diesel. It would be very difficult to fill a closed bucket. Try finding out what really happened before posting. Every container was closed securely. Mike, you are extremely clueless Everybody involved was aware that it was done to keep Peer1's generator running.