How to hunt small animals effectively?

If you have the default aim helper enabled, it's pretty easy to target the small animals, though the really tiny ones can be tricky.

For rabbits, just shoot them with a varmint Rifle. It only takes one shot and doesn't harm the quality. I simply shot at every rabbit I saw from my horse to fulfill the Horseman and the Master Hunter challenges about rabbits. If you have the Horseman challenge about killing 5 rabbits from your horse already, you can check the quality before shooting, simply targeting the rabbit with your Varmint Rifle will show if it's a three star or lower pelt.

I had issues with squirrels, the Varmint rifle seems to degrade pelt quality. Using small game arrows works, I'm not entirely sure if you have to hit them in the head or not.

I haven't used it myself for this purpose, but you can of course also use Dead Eye to help with killing animals.

Do four things: a) Play the main quest until you unlock the ability to manually use Dead Eye. Until this, every time your aim goes over an enemy, it will "paint" it with dead-eye shots. This ruins pelts b) Read the Compendium for the animal you're hunting - it will tell you what type of weapon(s) can be used c) When in Dead Eye, aim only for red-highlighted spots, or the head d) Hunt only 3-star animals (or 2-star animals if you have the Elk Trophy and are willing to only get a Good hide)