Outage Highlights Battery Maintenance

6 comments

On August 5 some customers at an Internap Network Services (INAP) data center in Boston lost connectivity for 1 hour, 20 minutes due to a battery failure in a DC power plant. 

“The DC plant was responsible for powering network equipment for both Internap and other carriers,” Internap said in a statement. “In some cases this network equipment was used by our customers for IP connectivity. The failure was caused by a utility company power interruption that cause the DC plant to fail over to battery backup. DC plants do not afford the ability to simulate load via a load bank as can be done on a UPS or generator. However, in this instance, when the batteries were called upon, they did not carry the load. The batteries had not shown any prior signs of pending failure during normal testing and are still being investigated at this time. Since the incident, the source of the power fed to the networking equipment was changed to UPS-backed AC power to avoid any potential repeat of this event.”

The incident underscores the importance of battery maintenance and monitoring. Mark Fontecchio at Tech Target has a story that looks at four ways that data center managers can extend the life of their UPS batteries.

About the Author

Rich Miller is the founder and editor at large of Data Center Knowledge, and has been reporting on the data center sector since 2000. He has tracked the growing impact of high-density computing on the power and cooling of data centers, and the resulting push for improved energy efficiency in these facilities.

Add Your Comments

  • (will not be published)

6 Comments

  1. Please explain why the batteries could not be load banked tested. DC load banks for -48VDC or -24VDC are commonly available. The article in Tech Target does not really apply to this case.

  2. George Pedersen

    Unlike a UPS that requires the load to be transferred off the UPS during a load test. Most of the DC plants installed in the last ten years have the abilty to reduce the output voltage of the rectifiers to below battery open circuit but above the drop out voltage of the power supplies. This allows the batteries to pick up the load and check their viability without risking the load. The comment about the batteries had not shown signs of pending failure during normal testing is regrettably quite normal. Batteries do frequently fail between scheduled maintenace visits which is why the majority of High Reliability data centers now incorporate on line battery monitoring.

  3. It seams unlikely that any power engineer would use a single string of batteries to support critical load. With multiple strings of batteries it is quite easy to pull one string off at a time for testing. IEEE 1188 and 450 provide great recommendations for testing sealed and flooded batteries. It would be interesting to see a single line and to find out if the policy at the location is to only monitor batteries as opposed to performing maintenance including a load test.

  4. It IS possible to test batteries while they are in service and WITHOUT a load bank. Telecom companies have been doing it for almost 100 years. You have to understand batteries and realize that just reading the voltage of a battery does NOT provide a true status of the battery. Never has, but VRLA battery suppliers would like you to think that is enough so that their batteries look more economical. True health in a battery plant is much more than just battery maintenance. You have to know that your AC feed will handle the surge presented when all of the rectifiers restart simultaneously, and you absolutely need 20% of capacity (or one additional rectifier) in the plant beyond the load in order to have enough power to recharge the batteries after an extended AC failure.

  5. I must take issue with this article. It is simply not true that one cannot test the battery in a DC power plant. In actual fact, it is a lot easier and safer to do this than test a UPS battery. My company is doing this for clients on a daily basis. Telecommunications companies have been doing it for almost a hundred years. The procedures are simple and are historically well defined. In many methods the battery does not even have to be taken off-line. Both IEEE 450 - Recommended Practice for Maintenance, Testing and Replacement of Vented Lead acid Batteries for Stationary Applications, and IEEE 1188 - Recommended Practice for Maintenance, Testing and Replacement of Valve-Regulated Lead-Acid (VRLA) Batteries for Stationary Applications, address the subject in depth. Years of operational records has proved that a properly designed battery backed DC power system is much more reliable than a UPS. Why do you think the telecommunications companies have always used this architecture? It is no accident that they have the most reliable systems and networks. The battery plant (batteries seem to get the blame for all system outages) in a DC system are usually designed for a minimum of four hours reserve and often features redundant battery strings and real time monitoring. A sudden battery failure should never happen. I suspect in the Internap case that either poor system design, lack of an adequate maintenance plan, poor maintenance or inadequate battery status trending data, or a combination of all of the above, caused the “battery failure.” Any battery system owner that truly cares about their battery reliability should have a proper battery maintenance and test program in place. All too often money is not a hurdle when it comes to the maintenance of other equipment in a data center or other critical location but when it comes to the humble battery this often ignored. A good reference document, which addresses all aspects of maintenance in simple terms, is IEEE 1491 - Guide for Selection and Use of Battery Monitoring Equipment in Stationary Applications.

  6. The proper testing and maintenance of your battery plant is crucial to its long term survival, access to the testing data is just as important. The ability to review your battery plant history is a key factor when faced with replacement or maintenance decision. BatteryNet achieves your maintenance records in a convenient location available for immediate review.