How can I stop playing nether portal tag with a mooshroom?

According to the Minecraft Tumblr, this has been fixed for 1.4.4. If you haven't updated or can't due to the server your on, or are still having problems there may still be some solutions.

One thing you can try that I've seen having at least limited success, is dig a pit around the portal in the nether, such that it would be impossible for the animal to jump back in. It looks like you may have tried this already, but I don't think it was deep enough (at least 2 blocks down from the edge of the portal).

If this is on a server, another option to at least try (I have no evidence that it'll work) is to have another player waiting on the other side with wheat (or another appropriate luring agent) so that the animal is lured away from the portal as soon as you teleport them.