Anyone leaving the Nether exits through my house?
Most likely you need to take the time to properly align the portals in both worlds. It takes some effort, but if you exactly align your portals everything seems to work correctly on the SMP server I regularly play on.
From the Minecraft wiki.
To setup pairs of portals properly so that they reliably travel to each other, it is best to build both portals manually. Build at desired location X,Y,Z in the Normal World. Then travel to the Nether World. And then dig your way to X/8, Y, Z/8, and build a portal there.