Xayah's Passive Damage

Solution 1:

After doing some research I stumbled upon this reddit thread which confirms my suspicions.

Xayah's autoattacks will still apply an on-hit effect to the her initial target, however all subsequent targets will take damage based on the incoming attack without on-hit effects.

As pointed out in the reddit thread her passive is more comparable to a Kayle E than to a Twitch R/Runaan's Hurricane.

In the case you've described above Xayah would deal 120 damage to the primary target and her feathers would deal 50 damage to subsequent targets.
Please note that the incoming attack could critically strike, resulting in the attack dealing 220 damage to the primary target but only 100 damage to the secondary target(s).

Solution 2:

According to http://leagueoflegends.wikia.com/wiki/Xayah (click show ability details)

On-hit effects are only applied to her selected attack target

BotRK's passive is an on-hit effect, so in your example, the target you attacked would take 100+20=120 damage, and other targets would take 100/2=50 damage. These numbers would also be reduces by the individual targets armor.

basically, the main target takes 100% AD + on-hit damage, and the other targets take 50% AD damge.