Why do only a few English demonyms indicate gender?

Spanish has a fairly simple system of indicating gender agreement that is familiar to a great many English speakers (maybe most), which means English speakers are more likely to be aware of how to inflect Spanish adjectives for gender than adjectives from most other languages.

Even though French gender agreement is also relatively simple—mostly a matter of adding "e"—the resulting forms would often be homophonous if read using English conventions, which I think leads to more confusion. The French-derived words that are inflected in writing in English, such as né(e) and fiancé(e), are fairly often misspelled.

Additionally, the word "Latino/Latina" entered English relatively recently, so it's had less time to lose foreign affixes and become naturalized. The earliest citations for Spaniard in the Oxford English Dictionary are from around the fifteenth century, while the earliest for Latino is 1946.

The continued use of these terms by bilingual speakers, as you have mentioned, is also likely to play a role in the maintenance of the gender distinctions. Pinoy and Pinay in particular seem to me to be used most often by Filipinos or people of Filipino ancestry, rather than by other English speakers.

-man and -woman are not like the -o and -a of Romance languages. They are not inflections. They are compounds, not suffixes, and they are not even strictly parallel. Man was originally a word for person (genderless) and is still found filling that role in many words, though it has also gained a common secondary meaning of male person.