Can I stop removing guns from my inventory while riding my horse?

Solution 1:

Arthur putting his guns away does not affect the loadout. They are two separate things.

There is currently an issue with Arthur not remembering his loadout. Some Reddit users have found that equipping the weapons from the blacksmith menu seems to make Arthur remember his loadout for longer (though still not permanently afaik).
I expect this behavior to be patched at some point, it seems to be a bug more than an intended feature.

Arthur putting away his guns it a matter of animation. It makes sense that Arthur doesn't want a heavy (mostly) metallic object smacking into his back as he's riding.
There have been cases where I see Arthur put away the guns on the horse, but when I dismount not too long after, the weapons are back on Arthur's back.

The "unequipping" feature/bug does happen from time to time, but it's not directly tied to Arthur taking his weapons off while riding.