"that" or "this"

Although the action may be taking place in front of the person asking the question, it is nevertheless taking place at a distance, both physically and situationally. It's the OTHER person who is performing the action, and so it's taking place 'over there', in a different personal space. Why are you doing this, on the other hand, would often refer to an action that affects both parties.

You could use either word in that context, although "that" sounds much more natural to me. And the meaning is slightly different.

Why are you doing that?

Would be taken to mean "Why are you doing that specific thing that you are doing?"

Why are you doing this?

Would be taken in a more general sense, as if the thing the person is doing is affecting an entire situation.

Let me illustrate with an example.

Suppose a wife walks in to find her husband with another woman. If she says Why are you doing that? it would be taken to mean "Why are doing those things with that woman?" Whereas if she says Why are you doing this? she probably means "Why are you acting this way, disregarding my feelings, and putting our entire marriage at risk?"