Gitlab showing 404 while running behind nginx reverse proxy, all within a docker network
As the title says, I'm trying to serve Gitlab through an nginx reverse proxy, with both programs being run in separate docker containers connected through a docker network. A picture as an example:
Linux Host
____________________________
| |
| Docker |
| __________________________|
| | |
| | Docker network (test-net)|
| | ________________________|
| | | |
| | | nginx gitlab | Only nginx has a port bound to the host (443).
| | | | | | | | TLS is terminated at nginx as well.
| | | | | --> | | | in my test, I have nginx running as localhost.
| | | |___| |___| | To access gitlab, hit https://localhost/git/
| | |________________________|
| |__________________________|
|____________________________|
nginx runs with this docker command:
docker run -dit --network=test-net --name=nginx -p 443:443 -v "$PWD/conf":/etc/nginx:ro nginx:alpine && docker logs -f nginx
nginx.conf
<Removed unnecessary config from here, very basic setup>
http {
keepalive_timeout 65;
server {
listen 443 ssl;
server_name localhost;
ssl_certificate localhost.crt;
ssl_certificate_key localhost.key;
ssl_protocols TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
location /git/ {
proxy_pass http://test/;
}
}
}
gitlab.rb
<only relevant parts added here>
external_url 'https://localhost'
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
"Host" => "$http_host_with_default",
"X-Real-IP" => "$remote_addr",
"X-Forwarded-For" => "$proxy_add_x_forwarded_for",
"X-Forwarded-Proto" => "http",
"Upgrade" => "$http_upgrade",
"X-Forwarded-Ssl" => "on",
"Connection" => "$connection_upgrade"
}
nginx['custom_error_pages'] = {
'404' => {
'title' => '404',
'header' => 'You\'ve been hit by !! You\'ve been struck by ! A false URL.',
'message' => 'Double check that URL! Is it correct?'
}
}
docker-compose.yml for gitlab:
version: '3.7'
services:
gitlab:
image: 'internal-docker-repo:1234/gitlab/gitlab-ce:11.8.3-ce.0'
restart: always
hostname: 'test'
container_name: test
volumes:
- './config:/etc/gitlab:rw'
networks:
- net
networks:
net:
external: true
name: test-net
Internally (to docker networks) nginx is known as nginx and gitlab is known as test. I have confirmed I can ping each container from inside the other, using their container names.
As it is now, it almost works. When I go to https://localhost/git/ on my linux host I get a 404 error page from gitlab, but no login screen.
I'm obviously missing something but I'm not sure what it is. It's hard for me to tell if it's an NGinx configuration issue or a Gitlab configuration issue.
Log output when I hit https://localhost/git/
nginx log output:
172.19.0.1 - - [07/Jan/2020:21:28:35 +0000] "GET /git/ HTTP/1.1" 404 2289 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
gitlab log output:
test | ==> /var/log/gitlab/nginx/gitlab_access.log <==
test | 172.19.0.3 - - [07/Jan/2020:21:28:35 +0000] "GET / HTTP/1.0" 404 2289 "" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
test |
test | ==> /var/log/gitlab/gitlab-workhorse/current <==
test | 2020-01-07_21:28:35.10649 test 127.0.0.1:0 - - [2020/01/07:21:28:35 +0000] "GET / HTTP/1.1" 404 3108 "" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0" 0.001
test |
Solution 1:
I think at least one of the problems (if not the most important) is the fact that gitlab expects https://localhost as the URL in your browser. Unfortunately, you're not giving it that, but https://localhost/git, although the request obviously reaches the backend (which is the gitlab container).
Changing the external url directive on the inner nginx to "https://localhost/git" might work as a quick fix. Obviously you need to reconfigure gitlab afterwards:
sudo gitlab-ctl reconfig
If I were you, though, I would use a dedicated domain for gitlab (so a separate server block in nginx), as this would be cleaner to my mind. Something to the effect of:
server {
listen 443 ssl;
server_name gitlab.example.com;
ssl_certificate localhost.crt;
ssl_certificate_key localhost.key;
ssl_protocols TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
location / {
proxy_pass http://test/;
}
}
And then change the external_url, of course, to gitlab.example.com.
Since you're using the localhost anyway, I don't think it would be much bother if you added this new domain in /etc/hosts : 127.0.0.1 gitlab.example.com
Solution 2:
Cause
The reason you can't get connected is because of the location in Nginx mismatch the external_url URL defined in gitlab.rb.
In your Nginx configuration, you specify location /git/, while in the Gitlab configuration, you use https://localhost. This will cause the error: when a client requests to access https://localhost/git it will always tail the "git/" at the end of URI, however, your Gitlab is configured to access via https://localhost, so it will never be able to find the webpage!
Solution:
As either change the location /git/ to location / or change the https://localhost to https://localhost/git.
Example
Here is my full configuration on my docker, you can take this as reference:
*Note: 172.16.0.10 is the IP of the Gitlab server, gitlab.drive.nr resolves the IP of the Nginx server
Nginx docker:
$ cat nginx-r.sh
sudo docker run -itd --name=nginx-r \
--network=testenv \
--ip 172.16.0.3 \
--dns 172.16.0.2 \
-h rproxy.drive.nr \
-p 80:80 -p 443:443 \
-v nginx-config:/etc/nginx \
-v nginx-certs:/etc/ssl/private \
nginx:alpine
Gitlab docker:
$ cat gitlab.sh
sudo docker run --detach \
--hostname gitlab.drive.nr \
--name gitlab \
--restart always \
--network testenv \
--ip 172.16.0.10 \
--dns 172.16.0.2 \
--volume gitlab-config:/etc/gitlab \
--volume gitlab-logs:/var/log/gitlab \
--volume gitlab-data:/var/opt/gitlab \
gitlab/gitlab-ce:latest
Nginx configurations:
# cat conf.d/sites-enabled/gitlab.conf
server {
listen 80;
location / {
return 301 https://172.16.0.10;
}
}
server {
listen 443 ssl;
server_name gitlab.drive.nr;
ssl_certificate /etc/ssl/private/gitlab.crt;
ssl_certificate_key /etc/ssl/private/gitlab.key;
ssl_session_cache builtin:1000 shared:SSL:10m;
location /git {
proxy_pass http://172.16.0.10;
}
}
Gitlab configurations:
# cat gitlab-config/gitlab.rb |grep -v ^#
external_url 'http://gitlab.drive.nr/git'
Test Drive
Log on Nginx:
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git HTTP/1.1" 302 102 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/users/sign_in HTTP/1.1" 302 150 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU HTTP/1.1" 200 9502 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/application-aeddf31361633b3d1196c6483f25c484855e0f243e7f7e62686a4de9e10ec03b.css HTTP/1.1" 200 149097 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/print-74c3df10dad473d66660c828e3aa54ca3bfeac6d8bb708643331403fe7211e60.css HTTP/1.1" 200 382 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/webpack/runtime.ee78bc38.bundle.js HTTP/1.1" 200 2134 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/highlight/themes/white-3144068cf4f603d290f553b653926358ddcd02493b9728f62417682657fc58c0.css HTTP/1.1" 200 864 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/webpack/main.b91d0a07.chunk.js HTTP/1.1" 200 830262 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/webpack/default.ca6f81b2.chunk.js HTTP/1.1" 200 159 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:27 +0000] "GET /git/assets/icons-e91700f3f1ecff110fc2c35aa62aec8f2aad69d1bfb35844186a11175a79e25f.svg HTTP/1.1" 200 25983 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:27 +0000] "GET /git/assets/touch-icon-ipad-retina-8ebe416f5313483d9c1bc772b5bbe03ecad52a54eba443e5215a22caed2a16a2.png HTTP/1.1" 200 5662 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:27 +0000] "GET /git/assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png HTTP/1.1" 200 1611 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"