How do VPN servers/clients avoid routing their own remote connection packets through themself?

You are correct that with purely destination-based routing this is a problem, if the destinations you are reaching through the tunnel overlap with routing needed for tunnel establishment, etc....

The way I have usually seen this done, and done it myself on various routers, is to use policy routing :

• The router acting as a VPN endpoint keeps its default route pointing to the Internet through its ISP link
• It also has a Policy Route with a source-based rule saying that traffic coming from the subnets behind it, whatever their destination, should be sent through the tunnel.

With destination based routing it's not hard either.

The way I usually see is to load a route to the VPN server specifying the pre-existing gateway and interface and distance (really routing priority) set to 1. The VPN's default route would always have a distance of at least 2.

Imagine a system with 1 physical adapter: enp2s0.

It's routing table might start out as all traffic goes out enp2s0.

Once the system connects to a VPN, a TUN interface (tun0) is initialized and the VPN updates the routing table: All traffic destined to VPN server address on port X (VPN server address and port) goes out enp2s0. All other traffic goes through tun0.

Of course a VPN doesn't have to route all traffic. For instance, I have set up VPNs before where only traffic that would go to a certain private subnet goes through the VPN so normal internet traffic would be unchanged. How you do this changes based on the VPN program.