What is the equivalent of --add-host=host.docker.internal:host-gateway in a Compose file

Starting from Docker version 20.10 (https://github.com/moby/moby/pull/40007), there is a new special string host-gateway that one can use within the --add-host run flag to allow a direct connection from inside a docker container to the local machine on Linux based systems. And this is very nice.

But what is the equivalent of --add-host=host.docker.internal:host-gateway in a Compose file?

e.g. in:

$ docker run \
  --rm \
  --name postgres \
  -p "5433:5432" \
  -e POSTGRES_PASSWORD=**** \
  --add-host=host.docker.internal:host-gateway \
  -d postgres:14.1-bullseye

How would the same --add-host flag fit in this Docker Compose equivalent template:

version: '3.9'

services:
  postgres:
    image: postgres:14.1-bullseye
    environment:
      POSTGRES_PASSWORD: ****
    ports:
      - "5433:5432"

It's for sure not: network_mode: host at the service level (see #Doc).

The actual Docker Compose equivalent is achieved by appending the same string to the extra_hosts parameters (#Doc) as:

version: '3.9'

services:
  postgres:
    image: postgres:14.1-bullseye
    environment:
      POSTGRES_PASSWORD: ****
    ports:
      - "5433:5432"
    extra_hosts:
      - "host.docker.internal:host-gateway"

You can see it has been successfully mapped to the IP of the docker0 interface, here 172.17.0.1, from inside your container, e.g.:

$ docker-compose up -d
$ docker-compose exec postgres bash

then, from inside the container:

root@5864db7d7fba:/# apt update && apt -y install netcat
root@5864db7d7fba:/# nc -vz host.docker.internal 80
Connection to host.docker.internal (172.17.0.1) 80 port [tcp/http] succeeded!

(assuming port 80 is not closed or constrained to the IP of the docker0 interface by a firewall on the host machine).

More on this can be found here:
https://medium.com/@TimvanBaarsen/how-to-connect-to-the-docker-host-from-inside-a-docker-container-112b4c71bc66

But... beware...

Warning ⚠️

This will normally always match the 172.17.0.1 IP of the docker0 interface on the host machine. Hence, if you spin-up a container using a Compose file (so, not by using docker run), chances are infinitely high that this container will rely on the network created during the build of the Compose services. And this network will use a random Gateway address of the form 172.xxx.0.1 which will for sure be different than the 172.17.0.1 default docker Gateway, this can for example be 172.22.0.1.

This can cause you some troubles if for example you only explicitly authorized connections from 172.17.0.1 to a port of a local service on the host machine. Indeed, it will not be possible to ping the port of that service from inside the container, precisely because of this differently assigned Gateway address (172.22.0.1).

Therefore, and because you cannot know in advance which Gateway address the Compose network will have, I highly recommend that you wisely build a custom network definition in the Compose file, e.g.:

version: '3.9'

networks:
  network1:
    name: my-network
    attachable: true
    ipam:
      driver: default
      config:
        - subnet: 172.18.0.0/16
          ip_range: 172.18.5.0/24
          gateway: 172.18.0.1

services:
  postgres:
    image: postgres:14.1-bullseye
    environment:
      POSTGRES_PASSWORD: ****
    ports:
      - "5433:5432"
    networks:
      - network1

If needed, I also recommend using some IP range calculator tool, such as http://jodies.de/ipcalc?host=172.18.5.0&mask1=24&mask2= to help yourself in that task, especially when defining ranges using the CIDR notation.

Finally, spin up your container. And verify that the newly specified Gateway address 172.18.0.1 has been correctly used:

$ docker inspect tmp_postgres_1  -f '{{range .NetworkSettings.Networks}}{{.Gateway}}{{end}}'
172.18.0.1

Attach to it, install netcat and verify:

root@9fe8de220d44:/# nc -vz 172.18.0.1 80
Connection to 172.18.0.1 80 port [tcp/http] succeeded!

(you may also need to adapt your firewall rules accordingly and/or the allowed IPs for your local service, e.g. a database)

Another solution

is to connect to the existing default bridge network using docker network. In order to do so, after having spin up the container, run this command:

$ docker network connect bridge tmp_postgres_1

Now, an inspect should give you two IPs; the one you set up (if any) or the one auto-magically set up by docker during the container creation, and the bridge IP:

$ docker inspect tmp_postgres_1 -f '{{range .NetworkSettings.Networks}}{{.Gateway}}{{end}}' 
172.17.0.1 172.18.0.1

Or

you can skip the manual network creation and directy tell, in your Compose service definition, to join the bridge network using the network_mode: flag as follow:

version: '3.9'

services:
  postgres:
    image: postgres:14.1-bullseye
    environment:
      POSTGRES_PASSWORD: ****
    ports:
      - "5433:5432"
    # removed networks: and add this:
    network_mode: bridge
    extra_hosts:
      - "host.docker.internal:host-gateway"

Now, whether you used the docker network connect... method or the network_mode: flag in your Compose file, you normally succesfully joined the default bridge network with the Gateway 172.17.0.1, this will allow you to use that Gateway IP to connect to your host, either by typing its numerical value, or if set, the variable host.docker.internal:

root@9fe8de220d44:/# nc -vz 172.18.0.1 80
Connection to 172.18.0.1 80 port [tcp/http] succeeded!
root@9fe8de220d44:/# nc -vz 172.17.0.1 80
Connection to 172.18.0.1 80 port [tcp/http] succeeded!
root@9fe8de220d44:/# nc -vz host.docker.internal 80
Connection to host.docker.internal (172.17.0.1) 80 port [tcp/http] succeeded!

⚠️ But by joining the bridge network, you also makes it possible for your container to communicate with all other containers on that network (if they have published ports), and vice-versa. So if you need to clearly keep it apart from these other containers, you preferably don't want to do that and stick with its own custom network!

What if something goes wrong?

In case you messed up your docker network after some trials, you may face such error message:

Creating tmp_postgres_1 ... error

ERROR: for tmp_postgres_1  Cannot start service postgres: failed to create endpoint tmp_postgres_1 on network bridge: network 895de42e2a0bdaab5423a6356a079fae55aae41ae268ee887ed214bd6fd88486 does not exist

ERROR: for postgress  Cannot start service postgres: failed to create endpoint tmp_postgres_1 on network bridge: network 895de42e2a0bdaab5423a6356a079fae55aae41ae268ee887ed214bd6fd88486 does not exist
ERROR: Encountered errors while bringing up the project.

even so the 895de42e2a0bdaab5423a6356a079fae55aae41ae268ee887ed214bd6fd88486 bridge network does actually exist, you have to clean all that either by restarting your computer or in the luckiest case, the docker service with:

$ sudo service docker restart

(a docker networkd prune -f may not be sufficient).

More in the documentation:
https://docs.docker.com/compose/networking/
https://docs.docker.com/compose/compose-file/compose-file-v3/#networks
https://github.com/compose-spec/compose-spec/blob/master/spec.md#networks-top-level-element

Tested on a host machine having the following specs:
Ubuntu: 18.04.6 LTS
Kernel: 5.4.0-94-generic
Docker: 20.10.12, build e91ed57
Docker Compose: 1.27.4, build 40524192