Why is 'lavender' used to reference homosexuality?

I am aware of Lavender Linguistics, which is a form of Polari and was used by gay men a lexicon 'used in the 1950s and early 1960s by gay men as a secret language which concealed their homosexuality in the face of the repressive society of the time.'

• Wiktionary doesn't offer this definition.
• Etymonline mentions 'camp followers' (which I presume means homosexuality) but not anything further than that.
• Lexico gives a definition of 'Used in reference to effeminacy or homosexuality.'.

I thought Why Lavender? - The University of Nottingham would help, but instead it speaks about the reason a conference uses the term lavender, reiterating its use in the 20th century to denote homosexual people:

‘Lavender’ was seen to be a neutral term which would not exclude particular research foci...

In short, I know that it is used in this way, but why did lavender become associated with homosexuality?

I assume it is because purple is regarded as feminine, but can't find anything to back it up.

Lavender, an adjective used to refer to homosexuals is from late 19th century according GDoS

(orig. US) a euph. for homosexual and anything referring to homosexuality; also as n., homosexuality.

• [1870 [UK] ‘The Ninety-Ninth Hussars’ in Songs for the Army 46: Sir Lavender Silk was a pretty young man, [...] / His men, though respectful, had thoughts of their own / Which might have spoke out if they chose, / That Sir Lavender Silk had the aspect alone / Of a Lady dressed up in men’s clothes!].

but usage examples become more common from the ‘20s

1928 [US] M. West Pleasure Man (1997) Act I: stanley: And don’t you annoy the boys, Violet. paradise: Lavender, maybe, but violet never.

According to the site out.com the lavender color was used in early 20th by lesbian women’s movement:

Bill Leap, perhaps the world’s most respected scholar in the field known as lavender linguistics, talks in a Southern drawl and cusses like a trucker’s wife.

In 1993, Leap created the Lavender Languages and Linguistics Conference, now in its 24th season. The two-day event draws about 150 attendees from all over the world and is the longest-running LGBT-studies conference in the U.S., and the only one dedicated to language issues, according to Leap. In 1993, much like today, the community squabbled over language politics, starting with what to call the field of study — queer language? Gay and lesbian language? Leap went with lavender.

“I thought, Let’s use that ancient term ‘lavender’ and let’s offend everybody,” he says. Lavender, he points out, has been associated with the occult and mysticism, with women’s power in Africa, and with forms of power in the West in the Roman Imperial Court and the Catholic Church.

“It surfaces in the 20th century with a lesbian women’s movement in England, which was marked in public by women who wore lavender-colored rhinoceros pins on their lapel,” he says.