Why and when did "breast" become gender-specific?

Solution 1:

In Old English the word bréost was considered a neuter gender word. Before that it was considered a dual and carried with it an implied feminine. You will see it spelled "brest" sometimes in old texts and that spelling carries the dual. Since the late 18th century or so, however, "breast" can be either neuter or feminine depending on context whereas "breasts" carries only the feminine.

Solution 2:

Breast can sometimes refer to the entire chest, not to individual mammary glands. Think of double-breasted suits, which are traditionally worn by men, not by women.

You also occasionally see it used this way specifically referring to the place that air comes from. Consider the red-breasted nuthatch, the rose-breasted grosbeak, or little robin redbreast himself.

Breast (in the singular, not the plural as with mammaries) is also used figuratively to mean the seat of emotions. Used in this way, it cannot be exchanged with “chest”. Consider the phrase “his breast swelled/swollen with pride”. You couldn’t say that “his *chest swelled with pride” instead, because that would be a purely physical thing, and the swelling is here figurative.

As for even purely physical breasts being unique to women, that is patently untrue. Otherwise how could there be any such thing as breast cancer in men? Since there sadly enough indeed is such a thing, it is clear that men must necessarliy also have breasts, even if these do not (well, seldom) lactate.

In summary, your question has a false premise in it, for men have breasts, too. Just ask any physician.

Solution 3:

Musick has Charms to sooth a savage Breast,

To soften Rocks, or bend a knotted Oak.

. . . [and]

last Night

The silent Tomb receiv'd the good Old King;

He and his Sorrows now are safely lodg'd

Within its cold, but hospitable Bosom.

-William Congreve, in The Mourning Bride, 1697

In neither case is Mr. Congreve just alluding to the feminine.