New UPS Batteries Dying in Months

This is something that has had me stumped for over a year. I have 3 APC SmartUPS 3000's and a few non APC units in my server room. Each unit has it's own dedicated connection to the breaker panel.

It seems that I am having to replace the batteries in these units once every 4 months or so. We rarely ever have a power outage, as in not a single outage since the last time I replaced the batteries. I thought maybe it was bad line voltage, so I logged the UPS's output and I did not have a single sag or over-voltage scenario in those 4 months.

Maybe the lines were wired wrong? I tested them and traced them back to the panel and they all checked out.

I decided that the only possibility was that the replacement batteries I was using were not that great (Duracell). I proceeded to buy two brand new APC UPS units with new APC batteries of course, and after four months they were both dead.

The units never register an event in which they switch to battery power, they just simply report bad batteries after 4-6 months. This is a recurring problem that affects multiple units of different models and different brands of batteries. I have no clue whats going on here, I've had electricians and the provider test for various things, and it always comes back clean.

At this point I can only think of something being wrong at the provider level (bad transformer?), but usually that will cause voltage issues. I have also heard of people having phase shifting problems, but I have single phase power, so I am not sure if that is relevant or not.

Note: I used to be an electrical engineer and I have a fairly high level of understanding on how this is supposed to work. So you don't have to dumb down your answers :)

Solution 1:

That is not well known feature of the APC Smart UPSes.

Each time your UPS goes back on the wall and the batteries are recharged, controller counts the energy stored in them. Acid accumulators have some voltage that means "fully charged". When that voltage is reached charging is stopped and new calculated value of the battery's capacity is stored in the controller's NVRAM. Under any circumstances controller do not overhead that value, even if the target voltage is not reached.

When UPS operate native batteries, their capacity is slowly degraded with time. Therefore charging always stopped on the voltage limit, not capacity value. But when you have replaced the batteries, USP knows nothing about that and used an old capacity value to the fresh batteries. That is why they are always charged like they are old and degraded, never reached the needed voltage.

Chemistry of the acid-lead accus require them to be fully charged each time, or, being undercharged regularly, they degraded very, very fast, exactly as you have seen.

APC provide battery replacement service that include reset of controller to the initial values. Sure, in that case controller deal with batteries properly.

There is two news, as usual.

Bad one is that APC's controllers have proprietary protocol and should be resetted by certified APC engineer.

Good one is that some guys have hacked APC's protocol and wrote the tiny tool UPSDiag that allows to connect to the controller and perform the reset. Also you SHOULD use the serial cable, not the USB one.

Unfortunately, APC protocol wasn't hacked completely, so you have to be a bit lucky to log into controller. Once I've try over 20 times until being logged in. Then you have to perform calibration. Capacity value initially is set to some really big number. Then UPS charges batteries until their voltage reach limit. Then UPS drain the batteries to the voltage equivalent to the 25% of capacity. And then charge them again to the full. You have to connect some load to the UPS for calibration, at least 40% of it's power. It is about 600W for Smart-1500, 1.2kW for Smart-3000 and so on.

During that cycle real actual capacity of batteries is calculated and stored in the NVRAM. Voila, you have properly replaced batteries.

By the way, after 5-10 cycles batteries significantly increases their capacity, as well as maximum discharging current. Capacity growth can be about 10-15% of the nominal capacity. So good practice is to recalibrate batteries again after month or two.

Solution 2:

My thought is that you should have an electrician check the quality of the power. I have this same issue in a number of the plants I support. Our issue is that the plants have dirty and uneven/low voltage due to our server rooms power not being separated from the manufacturing equipment. This causes the ups's to work hard to keep voltage steady and causes the batteries to fail one to two times a year at my worst plants.