Origin of "man!", "(oh) boy!", and "oh brother"
Solution 1:
These are all considered "minced oaths". "Oh, man" could be replaced one-to-one with a Commandment-breaking "Oh, God" or "Oh, Jesus", which are generally shortenings of some prayer for strength.
An alternate etymology is that of friendly condescension; the terms can be literally used when talking to one's peer, junior, or brother: "Oh boy, you've screwed up this time" could be used when talking to your son or one of his friends after they break a window. The terms have evolved out of their literal meaning to simply become an exclamation indicating exasperation at a situation, even when you're not really referring to anyone else in particular.
The gender bias of the interjections is largely reflective of the historical gender bias in English-speaking cultures throughout the evolution of the language. There has been some insertion of the fairer sex in interjections: "oh girl" and simply "girl" are heard in informal conversation nearly universally now, virtually always when speaking directly to a "girl". Even then, there's no small amount of chauvinism in the term; the word as an identifier for females in general stems from the general male preference for youthful-looking women.
Solution 2:
Remember that modern English rests on at least 400-years of patriarchy and male-dominated society. Mark Twain alone gave a huge chunk to our culture.
Plus - Fact is, it is easier to say "Aah man" than "Aaah woman" - An extra syllable does make a difference. But "aah lady" sounds K
But I'm all for fairness. Perhaps you and I can introduce some more gender neutrality? How about, "Gee willickers" or "Aaah Zebras!"
At BK the other day, I heard a young lady say "No pickles-No onions" very fast. and I mimed it like 50 times it was real fun to say.
Be creative and soon we'll pick it up.
Solution 3:
According to what I've read, "There’s no indication that these expressions began life as euphemisms" (i.e., minced oaths). Rather, they appear to have originated from noun references to the second person, being used in place of their name. Sometimes even today, you may speak to someone and say "I enjoyed seeing you, man." However, the terms gradually evolved into standalone interjections so that you can use the terms even when the one spoken to isn't male. Source: https://www.grammarphobia.com/blog/2011/03/man.html
As for why the interjections are primarily male, this is attributable to the same reason many gendered expressions are masculine by default. The source above does point out, however, that similar but different expressions are used in the feminine form, such as “way to go, woman” or “what’s happening, sister?” or “you go, girl.”