Is there any downside to not recruiting my friends?
According to this list of playable characters, the only real effect of not recruiting units comes from if you choose not to recruit Kliff or Faye in the beginning. If you neglect to do this, you can recruit them as Celica.
Other than that, the only other effect of not picking up optional characters is that you'll have that many less units in your army and you'll be unable to view their supports.