What is gender neutral for fraternity/sorority?

Solution 1:

Most co-ed fraternities are called "fraternities" (though I do know of one co-ed sorority; it was originally women-only, and retained the label "sorority" when it opened its doors to men).

However, there are some serious problems with your example:

• The phrase "fraternity of scholars" is already well-attested, and almost never refers to a Greek organization; rather, it's generally either figurative, or referring to co-ed groups like Phi Beta Kappa.
  • Note that Phi Beta Kappa, despite being a fraternity, and despite being named with Greek letters, is not a "Greek organization", and will not evoke the Animal House image that you seem to have in mind.
• Even without that phrase, when you talk about mathematicians forming a fraternity, people are likely to think of a professional fraternity. There are actually quite a few math fraternities, and most of them are co-ed, but (like Phi Beta Kappa) they are not "Greek organizations" and will not evoke a Animal House image. Even readers who are not familiar with any math fraternities, specifically, are likely to picture something along these lines: there exist chemistry fraternities, engineering fraternities, and so on, so a math fraternity is a natural idea.
• More generally — no co-ed fraternities are "Greek organizations". There's no gender-neutral way to evoke the Animal House image.
• A large proportion of Americans have belonged to fraternities and sororities that are Greek organizations — and the majority of these do not haze. You should consider whether you really want to turn off a good chunk of your readership by presupposing that they conform to a negative stereotype. (It's a bit like writing, "When women mathematicians get together, their catfights are […]" . . . no matter where you might go with it, you've lost a chunk of your audience. Though to be sure, a hazing joke will not be anywhere near as offensive as a catfight joke.)

If your goal is just to set up a joke about mathematics-themed hazing, I'd suggest writing something like, "When mathematicians haze each other, it's by […]". Your readers will get that it's a joke. It doesn't require lots of setup.

(By the way — "brotherhood" can also be gender-neutral, but it never conveys the specific idea of a Greek organization.)

Solution 2:

If it is literally a Greek organization on a college campus and you don't want to denote the gender, then I would say "Greek organization." If you are referring to a figurative brotherhood/sisterhood or a society of people whereby one can only join via invitation, I'd say "club" or "the society of (whatever joins that group of people together)."

Solution 3:

'brotherhood; is not gender neutral. perhaps only a woman could tell you this. nor is man or mankind. words that might suffice as gender neutral options for fraternity or sorority are: camaraderie, guild, affiliation, club, fellowship, house, kinship, order and society