A man's breast vs. a woman's breasts

Why is it that breast is used when referring to a man's chest, but breasts is used for a woman's? Could breast also mean a woman's chest, or do breasts have to be used when referring to a woman's chest?

It seems strange that we speak as if there are two things when referring to a woman's chest but speak as if there is only one when referring to a man's.

A women's chest is also called her breast but is typically only used with idioms:

She beat her breast.

I rest my head upon her breast.

Obviously, there is a bit of awkwardness regarding these sayings due to the specific meaning of individual breasts. But this actually isn't restricted to women either. A man also has two breasts that would correspond to each of his two nipples.

In short, the only difference between these words with regards to gender is that most people don't refer to male breasts — either as a whole or individually. This isn't a restriction of the words; people just don't make a big deal out of male breasts.

That being said, when they are being specifically referenced they are more commonly called pects or pectorals (which, again, is not a gender specific term) or the derogatory term man boobs which describes someone with sagging, overweight breasts.

The description of the meaning of breast given from the NOAD contains:

• either of the two soft, protruding organs on the upper front of a woman's body that secrete milk after pregnancy.
  • the corresponding less-developed part of a man's body
  • a person's chest