What is it called when the majority moves and leaves behind a small group of people effectively exiling those people but without forcing them to move
It seems you've shunned the three you left behind—you have deliberately avoided their company.
In AmE: You and your friends ditched the others at the table. (Can also be used for sneaking out of a restaurant and leaving your date at the table.)