Docker Swarm mesh routing doesn't work for independent subnets
Solution 1:
I made sure that needed ports are open;
but...
Tcpdump was on the manager side, worker's tcpdump was completely empty for both ports.
This is a sign the network is blocking the packets. There's more than one place to block the connections across the network, and only opening the ports on the hosts is often not enough. You'll need to identify the location the packets are getting blocked by checking each hop on the network with the owners of those devices.