Looking for an idiom opposite to "the more the merrier"

Solution 1:

As per my comment above, consider the fewer, the better fare.

The More The Merrier The larger the number involved, the better the occasion. For example, John's invited all his family to come along, and why not? The more the merrier. This expression was first recorded in 1530, when it was put as “The more the merrier; the fewer, the better fare” (meaning “with fewer there would be more to eat”), an observation that made its way into numerous proverb collections.

The American Heritage® Idioms Dictionary

Solution 2:

Two is a company three is a crowd may suggest the idea:

• said when two ​people are ​relaxed and ​enjoying each other's ​company but another ​person would make them ​feel less ​comfortable

you could use also the following, making it sound less serious:

better be alone than in bad company:

• It is better to be called a loner than to be labeled as a bad person. It is better off to be without any friends than to have friends with a bad character. You are known by the company you keep. So we must be careful while choosing our friends.

There is an interesting Italian proverb that says :

"the best wine is always in the small barrel" that suggests that the best things are found or done in small entities or but few people.