A generic word for any 'specialized genre'
Perhaps recherché: unusual and not understood by most people (Merriam- Webster).
This word has a spectrum of meaning, from "exquisite/choice", through "affected", to "pretentious/overblown".
Would you be happy with such ambivalence? I think, from your quote, you might...?
Obscure
I think the meaning is (a) well-known, and (b) has a wide application, so I will skip the definitions.
It is specifically applicable, given your description. The New American Dictionary states:
Obscure often expresses dissatisfaction at one's inability to identify something.
Two points:
genre itself is specific (or specialised1), and that you are going to give a [specific] genre is the expectation of the listener.
You are trying to identify a genre, but you can't. So you want to express all that in one word.
Consider:
Bill was a hipster who only enjoyed obscure music
Bill was a hipster who only enjoyed recherché music
The foreign word is foreign, unknown, obscure (sorry). You will have to stop and explain what it means to the other person.
Note
- Your title 'specialized genre' is redundant because genre is specific. Your description is completely understood.
As well as Margana's excellent answer, you could try arcane, recondite or esoteric.