Am I missing out on anything by staying faithful to my follower?

Solution 1:

There are some achievements linked to followers but nothing that requires to spend a minimum amount of time with them, at least for now.

Achievements that involve follower are:

A Guilding Light: Use the Templar as a follower.
Me, Myself and Eirena: Use the Enchantress as a follower.
Stolen Moments: Use the Scoundrel as a follower.
The Art of Conversation: Complete the main character, follower, and artisan conversation achievements
Friends with Benefits: Fully equip one of your followers.
For the Order: Listen to all Templar's conversations.
To Catch a Thief: Listen to all Scoundrel's conversations.
Secret of the Ages: Listen to all Enchantress's conversations.