Auxiliaries are difficult:

If I am not wrong, I see the problem is how you conjugate the auxiliary verb "to be/have".

The painful truth about auxiliaries is that they are difficult to use, and moreso to "play" with.

The verbs to do, to be, and to have make auxiliaries even more confusing, because as an auxiliary, they all share the same or similar meaning (learnenglish.de). What's even stranger is that their auxiliary usages are all conjugations of the word "to be" (en.bab.la) despite simultaneously being conjugations of their own verb.

Solution to your problem:

If you persist on using "have/had" conjugations, the past perfect passive alternative was already provided to you by Centaurus: "who(m) had [this] been followed by?" However, he pointed out that this is awkward and unpleasant to read

So, use a different conjugation. A good conjugation is was: "who(m) was [this] followed by?"

Alternatively, you can use a get construction: "By who(m) got [this] followed."