Why does my gun reload slowly sometimes even with Quick Eject Mag?

Sometimes when I reload, it's nice and quick, but the majority of the time, it's as slow as not having a Quick Eject Mag. Is taking damage the cause of the problem, or does it depend on if it's night and maybe I'm tired (although I'm not playing Survival and as far as I know none of the mods I have installed would slow down the reload speed)?

There is not a lot of information on the quick eject mag, but the effects are comparable to the Quick Hands perk.

If you look at the Quick Hands Wikia page there is a glitch that prevents it from working, occurring most frequently after entering or exiting Power Armor. Changing weapons seems to resolve the issue. It is possible the same glitch occurs while using the quick eject mag.

Other things to note is that neither the weapon mod nor the perk does not affect the time it takes to cycle the bolt on a bolt-action weapons such as the hunting rifle or pipe rifle. It only effects reload time. The bolt cycle time falls under the rate of fire category.

Almost all weapon animations are tied to the frame rate in Fallout 4. During moments of intense combat or explosions your frame rate may drop which will also affect the reload speed of your weapon (and rate of fire, and swing speed of melee weapons).

Lastly, some weapons have different and longer animations when reloading from an empty magazine than a partially depleted magazine. When reloading from a completely empty magazine your character may spend an extra moment chambering the first bullet into the gun's receiver. This is not necessary for a weapon that has more than one bullet left in the magazine