Using private physical networks with Docker swarm mode

I am working on a production setup with Docker in swarm mode (using Docker 17.03.1-ce). There will be 2 data centers involved. In both data centers, all machines have both a public IP and a private (local to data center) IP on a private network, so there will be 2 private networks.

Network traffic over the private network interfaces is free, while traffic over the public interface isn't after some limit (and it's slower), so where possible I would prefer network traffic to go over the private interfaces.

Now from what I understand (I think), all traffic between Docker nodes in swarm mode will go through the same network interface that's used for communicating with the swarm masters, which in my case will have to be the public ones to make multi DC networking possible. Most expected traffic however will be between nodes in the same DCs, and it would be very nice if Docker could somehow route traffic through my private networks if source and target nodes happen to be on the same private network.

I'm afraid this might not be possible out of the box because the swarm masters don't know about these private networks and the IPs the nodes have on them.

One solution I could think of is to set up a VPN and deploy the swarm on top of that, but this adds extra complexity and I'd prefer a pure Docker swarm solution.

Update: as suggested in a comment, the basis for a solution might be using iptables to route outgoing traffic to private IPs instead of public ones. However, if I would want to do this, my next problem would be how to manage all these rules. With 10 servers in a DC, I'd need 10 * 9 = 90 of them to route all the possible local traffic over the private network. I can imagine maybe some tool exists that could assist with such a task or I could create one, but maybe there's a much simpler way to do this.

Solution 1:

(Probably it should be a comment, but cannot comment yet)

As you cannot use hostnames and/or DNS to bootstrap the cluster, I see no way to force the cluster exchange raft data on the correct unmetered network, but I see you can use an interface name. Curiously, it is not stated on the swarm docs but an issue show the error message expects an IP or interface.

I wonder if you can set the cluster members to advertise using the private interface name, so that you get the most traffic in a the way you want.

Cannot test myself right now, but will next week as I may run into a similar issue for a upcoming project.