Do you earn less XP per character with a larger party?
As of Patch 1.03, you gain an additional 10% XP for every empty slot in your party. (i.e. 5 party members = +10% XP, Solo = +50% XP, etc.)
Keep in mind that there is a level cap, and substantially more XP in the game than is needed to reach it. Keep in mind also that the higher level you are when recruiting NPC's, the more decisions in their development will have been made - often poorly.