Solution 1:

In modern use in American English, the term ma'am has gained quite a bit more use than madam:

http://ngrams.googlelabs.com/chart?content=madam%2Cma%20%27%20am&corpus=5&smoothing=3&year_start=1950&year_end=2008

In modern use in British English, madam is slightly more popular than ma'am:

http://ngrams.googlelabs.com/chart?content=madam%2Cma%20%27%20am&corpus=6&smoothing=3&year_start=1950&year_end=2008

As a native American English speaker, madam seems a bit archaic but does not necessarily connote a tie with a brothel unless you refer to someone as a madam. For example, the Oxford English dictionary provides the following example for madam of a brothel:

1959 N. Mailer Advts. for Myself (1961) 279 A rather remarkable woman who had been the madam of a whorehouse.

But as a form of address, it is used differently:

1956 N. Algren Walk on Wild Side ii. 122 It's not a pot, Madam. And it's strictly not for sale.

The typical terms I've heard are miss for younger females and ma'am for older ones. You could potentially refer to someone as madam or ma'am in either spoken or written communication. In formal writing, for example to someone whose name you do not know, use madam in both cases. For example:

When addressing a letter to the holder of a particular position without knowing the name or gender of the addressee, it is common to write “Dear Sir or Madam,” (or in the United States, “Dear Sir or Madam:”

This holds in both American and British English.

In less formal writing or speech, I would suggest using whichever term is more popular for the community you are in--ma'am in American English, and madam in British English. In both, madam will seem a bit more formal.

Solution 2:

Maybe, just once, someone will call me "Sir" without adding, "You're making a scene." -- Homer Simpson

"Madam" as a noun certainly means "brothel keeper" in the US but I don't think that has stained its use as an honorific. I don't know why not; certainly no one whose job involves hooking things (like fish) or stripping things (like furniture) would be willing to describe that job as "hooking" or "stripping".

However, in the US, the honorific "Madam" is associated with a level of formality so great that anyone thus addressed will likely believe that she is being mocked (and usually, she'll be right).

In several Asian countries, the English word "hostess" is used to mean "madam" (in the improper sense); I've seen more than one party thrown in the US by new arrivals from the East almost go very very wrong when an American guest complimented the hostess using that word.

Solution 3:

This appears to be more a case of register. "Madam" is the female equivalent for "Sir." It's pretty much at the apex of formality.

"Ma'am," on the other hand, is more akin to "Mister" for men. A median level of decorum. Only context for "Madam" would risk confusion with the brothel-keeper. For example: When I've been lucky enough to dine at 4-star restaurants, women in the party are usually addressed as "Madam" by the waitstaff, and no offense is taken.

Solution 4:

Ngrams only show the usage in literature - which can be a bit selective.

Ma'am is used where you would use 'sir',for senior officer ranks, in the British police and armed forces.

Not sure what you would call a knighted owner of a brothel in BE - possibly "Madam ma'am" ?