IPv6 does not work in Docker swarm?

I have an Ubuntu server 16.04 LTS running both IPv4 and IPv6 (dual-stack). This server is also running Docker. However, I have problems whenever I try to access the containers in the swarm, but only over IPv6. Here is my steps:

I have created an simple app running a gnuicron webserver:

def app(environ, start_response):
    """Simplest possible application object"""
    data = b'Hello, World!\n'
    status = '200 OK'
    response_headers = [
        ('Content-type', 'text/plain'),
        ('Content-Length', str(len(data)))
    ]
    start_response(status, response_headers)
return iter([data])

I run this with gunicorn -w 4 -b [::]:5678 myapp:app on my server, and verify it works by running this on a dual-stack client in my network:

$ curl --connect-timeout 15 http://[2001:db8:db0::5]:5678
Hello, World!
$ curl --connect-timeout 15 http://192.168.10.5:5678
Hello, World!

I write a Dockerfile and run it:

docker build -t docker_ipv6_test .
docker run --rm --init -p 5678:5678 docker_ipv6_test

I run the same verification as about, and it works as expected. Then I write a docker-compose.yml:

version: '3'
services:
  ipv6_test:
    image: docker_ipv6_test:latest
    deploy:
      replicas: 1
      restart_policy: 
        condition: on-failure
      ports:
        - '5678:5678/tcp'

and run it:

docker stack deploy -c docker-compose.yml ipv6_test

Then this happens:

$ curl --connect-timeout 15 http://[2001:db8:db0::5]:5678
curl: (28) Operation timed out after 0 milliseconds with 0 out of 0 bytes received
$ curl --connect-timeout 15 http://192.168.10.5:5678
Hello, World!

But I was wondering if I am doing something wrong, or I have hit a bug?

All of my code and files are available here: https://github.com/SitronNO/docker_ipv6_test

Solution 1:

Swarm mode doesn't appear to support IPv6 with the ingress/service-mesh networking. There's an open issue on this that I'd recommend subscribing to and adding your thumbs up to get more attention.

As a workaround, services within the cluster can talk over IPv4, and you can publish the ports in host mode through a proxy running in global mode. E.g. here's a partial compose file:

version: '3.8'
services:
  proxy:
    deploy:
      mode: global
    ports:
    - target: 80
      published: 80
      protocol: "tcp"
      mode: "host"
    # ...

Solution 2:

You've hit a bug and done something wrong. It would appear from many github issues and the comment on https://docs.docker.com/compose/compose-file/#ipv4_address-ipv6_address that ipv6 just doesn't currently work in swarms:

If IPv6 addressing is desired, the enable_ipv6 option must be set, and you must use a version 2.x Compose file. IPv6 options do not currently work in swarm mode.

you may like to try IPv6 NAT https://github.com/robbertkl/docker-ipv6nat but that seems to be tricky to set up too, especially if you don't have ipv6tables etc.

What you've done wrong is not set up an ipv6 enabled docker swarm scope bridge network, OR start docker with ipv6 enabled - but considering the current documentation, that's not surprising.

-I would suggest no one has answered this question because there isn't a good answer yet.-

BUT one answer is let docker do what it does best and let the rest of linux do the rest. Leave the docker swarm as a purely ipv4 network and just forward ipv6 traffic to that network. That is not possible with ip tables forward from ipv4 to ipv6 but it is possible using socat e.g.:

sudo socat TCP6-LISTEN:80,,su=nobody,fork,reuseaddr TCP4:127.0.0.1:81

Note I moved the port docker is listening on to port 81, because although the docker_gwbridge for the swarm may not have ipv6 'enabled' it will still bind to the ipv6:80 port (sigh). Note socat will now forward BOTH ipv4 and ipv6 traffic to ipv4 port 80.

you can sudo nohup socat ... & that command to get it into the background.

see How to transparently tunnel a port from IPv4 to a remote IPv6 device? et al.