What are the exact mechanics for Bane's nightmare skill?

Bane's nightmare skill can work on both allies and enemies, and is transferred when the sleeping target is attacked, but what are the conditions for doing so? Does the attack have to be a targeted attack, or can any aoe effect cause the transfer (such as ancient apparition's ult)? Also, is it possible to deny allies with this spell, will they take damage while sleeping?

Yes, Bane Nightmare's skill can be used to deny allies.

Nightmare can damage allies (and Bane), and can thus be used to deny hero kills.

reference

Regarding the damage required to wake up the target can be any type: AoE, target skill or auto-attack, even Radiance can do it. This is the reason why playing Bane within low pub teams can be difficult sometimes.

Transferring the Nightmare effect to the attacker only works with auto-attack instead.

to add some information: manual orb effects like huskars/drows arrows and viper/obsidians attacks wake up the target but if they cast it manually (clicking the spell and then target the sleeping enemy) the attacker wont get nightmare.