How make openvpn work with docker
Solution 1:
Solution (TL;DR;)
Create /etc/openvpn/fix-routes.sh
script with following contents:
#!/bin/sh
echo "Adding default route to $route_vpn_gateway with /0 mask..."
ip route add default via $route_vpn_gateway
echo "Removing /1 routes..."
ip route del 0.0.0.0/1 via $route_vpn_gateway
ip route del 128.0.0.0/1 via $route_vpn_gateway
Add executable bit to the file: chmod o+x /etc/openvpn/fix-routes.sh
. Change owner of this file to root: chown root:root /etc/openvpn/fix-routes.sh
.
Add to your config following two lines:
script-security 2
route-up /etc/openvpn/fix-routes.sh
Explanation
Openvpn adds routes that for following networks: 0.0.0.0/1
and 128.0.0.0/1
(these routes cover entire IP range), and docker can't find range of IP addresses to create it's own private network.
You need to add a default route (to route everything through openvpn) and disable these two specific routes. fix-routes
script does that.
This script is called after openvpn adds its own routes. To execute scripts you'll need to set script-security
to 2
which allows execution of bash scripts from openvpn context.
Thanks
I'd like to thank author of this comment on github, also thanks to ovpn support.
Solution 2:
You can also get docker-compose working if you define the subnet CIDR in your docker compose file:
networks:
your-network:
ipam:
config:
- subnet: 172.16.238.0/24
gateway: 172.16.238.1
Another option: create first the network with the subnet CIDR and then specify in the docker compose file that you want to use this network:
docker network create your-network --subnet 172.24.24.0/24
In your docker compose file:
networks:
your-network:
external: true
Solution 3:
Based on answer from Anas El Barkani, here's a complete step-by-step example using PostgreSQL.
While VPN is not connected, create a permanent docker network:
docker network create my-network --subnet 172.24.24.0/24
In docker-compose file, specify network as external:
version: "2"
services:
postgres:
container_name: postgres
image: postgres
volumes:
- ./volumes/postgres/data:/var/lib/postgresql/data
environment:
- POSTGRES_DB=dummy
- POSTGRES_USER=user
- POSTGRES_PASSWORD=123456
- POSTGRES_HOST=localhost
networks:
- default
ports:
- "127.0.0.1:5432:5432"
networks:
default:
external:
name: my-network
That's all. Now you can enable your VPN, and start/stop container as usual:
docker-compose up -d
docker-compose down
No need to turn VPN on/off every time, or to add weird scripts as root.
Solution 4:
Disclaimer:
This solution originally designed for next configuration:
- Ubuntu 18.04
- OpenVPN 2.4.4
- Docker-CE 19.03.5
- Docker-Compose 1.24.0
- Bridge IPV4 networks
- Docker-Swarm not used
and may differ for other configurations.
Problem
Start your VPN connection.
Case 1
When you try to restart docker daemon you'll get in the logs:
failed to start daemon: Error initializing network controller: list bridge addresses failed: PredefinedLocalScopeDefaultNetworks
Case 2
When you try to create bridge network (implicitly docker
and docker-compose
try to create this kind of network) within next cases:
-
docker create network
without defining subnet parameter -
docker-compose up
without defining subnet parameter
you'll get:
ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network
Solution (TL;BPR)
Select address range for docker network from the private address space that not planned be to use for resources inside your VPN. Imagine that it is
172.26.0.0/16
.-
Add changes to the Docker's daemon config file
daemon.json
file. :{ "bip": "172.26.0.1/17", "fixed-cidr": "172.26.0.0/17", "default-address-pools" : [ { "base" : "172.26.128.0/17", "size" : 24 } ] }
Where:
-
bip
- aka «bridge ip»: specific bridge IP address for thedocker0
bridge network, which used by default if other was not specified. -
fixed-cidr
- CIDR range fordocker0
interface and local containers. Only needed if you want to restrict the IP range defined bybip
. -
default-address-pools
- CIDR range fordocker_gwbridge
(needed fordocker-swarm
) interface and bridge networks.size
parameter set the default submask for newly created networks inside this range.
We divide our initial
172.26.0.0/16
range by equal172.26.0.0 - 172.26.127.255
and172.26.128.0 - 172.26.255.255
pools in this example.Be careful with
daemon.json
formating, otherwise you'd get the error like this when restart docker's daemonunable to configure the Docker daemon with file /etc/docker/daemon.json
-
- Run your VPN connection
- Find your device name by running the comand. Usually it something like
tun0
ip addr show type tun
- Show created routes
ip route show dev tun0
- Find the pool that overlaps with our selected address, let it be:
172.16.0.0/12 via 10.8.0.1
-
Split that pool on subnets on chunks with our selected Docker pool
172.26.0.0/16
. You can use this amazing calculator by David C. We've got:172.16.0.1/13 172.24.0.1/15 172.26.0.0/16 172.27.0.1/16 172.28.0.1/14
-
Create
/etc/openvpn/mynetwork-route-up.sh
script, for OpenVPN for excluding our subnet from routes, with following contents (note that we excluded our network):#!/usr/bin/env bash echo "Remove the route that conflicts with the Docker's subnet" ip route del 172.16.0.0/12 via $route_vpn_gateway echo "Bring back routes that don't intersect" ip route add 172.16.0.0/13 via $route_vpn_gateway dev $dev ip route add 172.24.0.0/15 via $route_vpn_gateway dev $dev ip route add 172.27.0.0/16 via $route_vpn_gateway dev $dev ip route add 172.28.0.0/14 via $route_vpn_gateway dev $dev
-
Create
/etc/openvpn/mynetwork-route-pre-down.sh
script with following contents (note that we excluded our network):#!/usr/bin/env bash echo "Remove manually created routes" ip route del 172.16.0.0/13 dev $dev ip route del 172.24.0.0/15 dev $dev ip route del 172.27.0.0/16 dev $dev ip route del 172.28.0.0/14 dev $dev echo "Creating original route because OpenVPN will try to del that" ip route add 172.16.0.0/12 via $route_vpn_gateway dev $dev
-
Make that scripts executable
sudo chmod u+x /etc/openvpn/mynetwork-route-up.sh sudo chmod u+x /etc/openvpn/mynetwork-route-pre-down.sh
-
Add this lines to the end of your
.ovpn
configscript-security 2 route-up /etc/openvpn/mynetwork-route-up.sh route-pre-down /etc/openvpn/mynetwork-route-pre-down.sh
Restart your OpenVPN
Run (for removing networks that may conflict when daemon will restart)
docker network prune
- Restart Docker daemon
sudo service docker restart
Cause
OpenVPN used frequently to route all traffic through tunnel, or at least, proxy private pools. So why docker fails, when it started?
Case 1
When you starting the Docker daemon, it checks daemon's config bridge network for overlaping with routes (up->down stacktrace):
- docker-ce/components/engine/daemon/daemon.go:NewDaemon
- docker-ce/components/engine/daemon/daemon.go:restore
- docker-ce/components/engine/daemon/daemon_unix.go:initNetworkController
- docker-ce/components/engine/daemon/daemon_unix.go:initBridgeDriver
- docker-ce/components/engine/vendor/github.com/docker/libnetwork/netutils/utils_linux.go:ElectInterfaceAddresses
- docker-ce/components/engine/vendor/github.com/docker/libnetwork/netutils/utils_linux.go:FindAvailableNetwork
- docker-ce/components/engine/vendor/github.com/docker/libnetwork/netutils/utils_linux.go:CheckRouteOverlaps
- docker-ce/components/engine/vendor/github.com/docker/libnetwork/netutils/utils.go
- 💥💥💥
As you can see here, you can also disable creating the default bridge network in the daemon config for fixing this error.
Case 2
When the Docker's component libnetwork
try to create it's network, it check all available addresses for overlaping with routes. If nothing found, it returns an error (up->down stacktrace):
- [docker-ce/components/engine/daemon/network.go:createNetwork]
- docker-ce/components/engine/vendor/github.com/docker/libnetwork/controller.go:NewNetwork
- docker-ce/components/engine/vendor/github.com/docker/libnetwork/network.go:ipamAllocate
- docker-ce/components/engine/vendor/github.com/docker/libnetwork/network.go:ipamAllocateVersion
- docker-ce/components/engine/vendor/github.com/docker/libnetwork/network.go:requestPoolHelper
- docker-ce/components/engine/vendor/github.com/docker/libnetwork/ipam/allocator.go:RequestPool
- docker-ce/components/engine/vendor/github.com/docker/libnetwork/ipam/allocator.go:getPredefinedPool
- 💥💥💥
Of course, other scenarios to get this errors exists too. Gotta Catch 'Em All!
Workarounds (not recommended)
Create network with the subnet parameter
Docker allow you to pass subnet address range explicitly and it seems that doesn't perform an overlap check in this case.
https://github.com/docker/libnetwork/blob/922cd533eac14b6e0754756c5cacf9f44af5d699/network.go#L1657
Create a network when OpenVPN stopped, then start it
I don't dive deep in that, but I think, OpenVPN doesn't check overlapping.
P.S.
Thanks (https://stackoverflow.com/users/7918/jb)[jb] for his great answer, it inspire me a lot for wrtiting this answer.
For in-depth network understanding of Docker, you can read this articles:
- «Configure Custom CIDR Ranges in Docker EE» by Dave Thompson
- «Plan your installation» by Docker
Also, don't forget this!
Solution 5:
The default routes which cause the problem are pushed to the OpenVPN client by the OpenVPN server.
Rather then create a script to delete the routes you can simply stop the problematic routes from being created in the first place.
There are two ways to do this:
-
If you can change the settings on the OpenVPN server, edit the config and remove the
redirect-gateway
option. On my EdgeRouter the relevant line looks like this:openvpn-option "--push redirect-gateway def1"
One a Linux server I believe it would look like this:
push "redirect-gateway def1"
-
If you can't change settings on the OpenVPN server you can tell your OpenVPN client to ignore the pushed routes from the server. On my Linux client the relevant line looks like this:
pull-filter ignore redirect-gateway
Once you've made those changes and restarted the OpenVPN service you should be able to start Docker containers without the dreaded could not find an available, non-overlapping IPv4 address
error.