What are the downsides to giving away Migration Access?

I've been playing my first game of Stellaris as a Xenophile empire, and since I didn't know what the pros and cons were I have been giving away Migration Access to other empires whenever the option comes up. Now I'm a bit further in, it seems to me that it provides a lot of benefits:

• You get population on your planets faster due to immigration (as long as they have compatible terrain) and you get to keep the generated resources.
• You "take away" population from other empires which weakens them.
• You can build colony ships using any of the race on your planets, which makes it easier to colonize other planets. This works best with Gaia planets since a variety of races will want to migrate to those.

I can't think of any downsides, but I'm guessing there must be some since I sometimes get a bartering advantage just for offering Migration Access. What am I missing?

Should the original faction declare war on you in the future, they will require less warscore to take over any planets containing "their" pops in the future, as well as an easier time "digesting" the planet after conquest. (Source: Dev forum posts here, and here)

Further downsides include:

Increased unity costs for having non-native pops in your empire

Your pops will sometimes emigrate away to other planets

Sometimes your pops are just better than theirs - you probably don't want to trade your pop that's intelligent for one who is, say, enduring and rapid breeder in the late game