Docker Nginx reverse proxy configuration
Context:
I'm fairly new to nginx, and figuring out how to incrementally make this work. I've worked around this problem for a while till now, where it's actually hindering development on one of my apps as a result of improper behavior (headers not being passed between each host, request bodies being empty, etc).
I have three apps in php-fpm containers, and an nginx server in its own container. The nginx container acts as a single access point for serving requests to the php-fpm containers based on the requested hostname (basic vhost configuration). Everything works fine when using try_files, but when switching to proxy_pass the problem presents itself.
The nginx configurations are nearly identical for each of the php-fpm hosts, with the exception of the server_name being changed to match whichever container the request is supposed to be processed by.
My research resources are linked at the bottom of the page.
Problem:
This is where I'm confused. Depending on what I set proxy_pass to will change the error I see in the nginx logs. Under no circumstance when using proxy_pass will requests actually complete successfully. Here are the various settings with their associated error:
Setting:
proxy_pass http://appX/; # upstream name
Error:
nginx | 2018/05/08 22:19:48 [error] 10224#10224: *8442 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.18.0.1, server: appX.local, request: "GET / HTTP/1.1", upstream: "http://172.18.0.3:9000/", host: "appX.local"
nginx | 172.18.0.1 - - [08/May/2018:22:19:48 +0000] "GET / HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "-"
Setting:
proxy_pass https://appX/; # upstream name
Error:
nginx | 172.18.0.1 - - [08/May/2018:22:20:50 +0000] "GET / HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "-"
nginx | 2018/05/08 22:20:50 [error] 13589#13589: *8446 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: 172.18.0.1, server: appX.local, request: "GET / HTTP/1.1", upstream: "https://172.18.0.3:9000/", host: "appX.local"
Setting:
proxy_pass http://appX.local/; # hostname
Error (Loops a bunch of times (logically, based on the redirect to https):
nginx | 172.18.0.6 - - [08/May/2018:22:21:49 +0000] "GET / HTTP/1.0" 301 185 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "172.18.0.1"
Setting:
proxy_pass https://appX.local/; # hostname
Error (Loops a bunch of times and I don't know why):
nginx | 172.18.0.1 - - [08/May/2018:22:23:35 +0000] "GET / HTTP/1.1" 500 588 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "-"
Setting:
proxy_pass https://appX:9000/;
Error:
nginx | 2018/05/08 22:29:26 [error] 14426#14426: *9563 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: appX.local, request: "GET / HTTP/1.1", upstream: "https://172.18.0.6:9000/", host: "appX.local"
nginx | 172.18.0.1 - - [08/May/2018:22:29:26 +0000] "GET / HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "-"
Setting:
proxy_pass https://appX.local:9000/;
Error:
nginx | 2018/05/08 22:27:39 [error] 14426#14426: *9559 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: appX.local, request: "GET / HTTP/1.1", upstream: "https://172.18.0.6:9000/", host: "appX.local"
nginx | 172.18.0.1 - - [08/May/2018:22:27:39 +0000] "GET / HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "-"
Attempted Remedies:
- I've tried changing the ports in the server directives to non-standard ports as well as appending a port number (both the one specified in the listen directive, as well as the one specified in the upstream directive).
- I've tried not using SSL and making this work over a standard HTTP connection.
- I've removed all of the other proxy settings and tried with just
proxy_pass - This works with
try_files $uri $uri/ /index.php?$args, but that bypasses the proxying. - I've tried to match the various configurations in my research resources and am experiencing the errors regardless of the changes I make.
Configurations:
Hostname are in my host's /etc/hostname file and all respond as to be expected.
nginx.conf
worker_processes 1;
daemon off;
events {
worker_connections 1024;
}
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
http {
default_type application/octet-stream;
include /etc/nginx/conf/mime.types;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
/etc/nginx/conf.d/appX.conf:
upstream appX {
server appX:9000;
}
server {
listen 80;
listen [::]:80;
server_name appX.local appX;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_tokens off;
ssl on;
ssl_certificate /etc/nginx/certs/appX.crt;
ssl_certificate_key /etc/nginx/certs/appX.key;
ssl_dhparam /etc/nginx/certs/dhparam.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
server_name appX.local appX;
root /var/www/appX;
index index.php index.html;
location / {
proxy_read_timeout 90;
proxy_connect_timeout 90;
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header Authorization $http_authorization;
proxy_pass_header Authorization;
proxy_next_upstream error timeout invalid_header;
proxy_hide_header X-Powered-By;
proxy_hide_header X-Pingback;
proxy_hide_header Link;
proxy_cache_bypass $http_pragma $http_authorization;
proxy_ssl_session_reuse off;
proxy_ssl_server_name on;
proxy_pass https://appX;
}
}
docker-compose.yml
version: '3.3'
services:
nginx:
image: evild/alpine-nginx:1.9.15-openssl
container_name: apps_nginx
volumes:
- ./app-one:/var/www/app-one/:ro
- ./app-two:/var/www/app-two/:ro
- ./app-three:/var/www/app-three/:ro
- ./nginx/conf/nginx.conf:/etc/nginx/conf/default.conf:ro
- ./nginx/conf.d:/etc/nginx/conf.d:ro
- ./certs:/etc/nginx/certs
ports:
- 80:80
- 443:443
expose:
- "80"
- "443"
depends_on:
- php-app-one
- php-app-two
environment:
TZ: "America/Los_Angeles"
networks:
default:
aliases:
- app-one.local
- app-two.local
php-app-one:
environment:
TZ: "America/Los_Angeles"
image: joebubna/php
container_name: apps_php_app-one
restart: always
volumes:
- ./app-one:/var/www/app-one
ports:
- 9001:9000
networks:
- default
php-app-two:
environment:
TZ: "America/Los_Angeles"
image: joebubna/php
container_name: apps_php_app-two
restart: always
volumes:
- ./app-two:/var/www/app-two
ports:
- 9000:9000
networks:
- default
php-app-three:
image: joebubna/php
container_name: apps_php_app-three
restart: always
volumes:
- ./app-three:/var/www/app-three
- ./fastcgi_params:/var/www/fastcgi_params
ports:
- 9002:9000
depends_on:
- db
environment:
WORDPRESS_DB_HOST: db:3306
WORDPRESS_DB_USER: root
WORDPRESS_DB_PASSWORD: root
WORDPRESS_DB_NAME: app_three
TZ: "America/Los_Angeles"
networks:
- default
db:
image: mysql:5.6
container_name: apps_mysql
volumes:
- db-data:/var/lib/mysql
- ./mysql/my.cnf:/etc/mysql/conf.d/ZZ-apps.cnf:ro
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_USER: user
MYSQL_PASSWORD: password
MYSQL_DATABASE: cora
TZ: "America/Los_Angeles"
ports:
- 3306:3306
expose:
- "3306"
networks:
- default
volumes:
db-data:
networks:
default:
driver: bridge
Any help would be greatly appreciated. I've spent quite a few hours on this today and I'm pretty well stuck as to what I've missed in all the reading and trial and error.
Research resources:
https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/
https://www.liaohuqiu.net/posts/nginx-proxy-pass/
https://gist.github.com/soheilhy/8b94347ff8336d971ad0
https://stackoverflow.com/questions/43500469/nginx-proxy-jwilder-nginx-proxy-connection-reset-by-peer-502-bad-gateway
https://groups.google.com/forum/#!topic/xnat_discussion/-S6eYsnwrXg
https://stackoverflow.com/questions/37346560/nginx-server-closes-ssl-connection-for-some-clients
https://stackoverflow.com/questions/38375588/nginx-reverse-proxy-to-heroku-fails-ssl-handshake
https://stackoverflow.com/questions/40580617/nginx-reverse-proxy-upstream-not-working
https://umbrella.cisco.com/blog/2015/11/03/lets-talk-about-proxies-pt-2-nginx-as-a-forward-http-proxy/
Use Nginx as Reverse Proxy for multiple servers
EDIT: Based on input from a couple kind individuals here, I have concluded this route has deviated too far from my original problem to warrant further pursuit. This can be chalked up to confusion and too many hours staring at the same problem. If a moderator wants to close this question, please feel free to do so.
The hostnames for docker are not specified in /etc/hostname or /etc/hosts. Docker has its own inter container communication. So when you refer to a container then you can use the service name that you specified in your docker-compose.yml to refer to the container. So at the moment you have the following containers:
- nginx: you can refer to it as nginx as the hostname. You should be able to ping the container using the hostname nginx.
- php-app-one: hostname: php-app-one
- php-app-two
- php-app-three
- db
Btw if you want to specify a different hostname for the container then you can use this in your docker-compose.yml
container_name: 'some-container-name-different-to-the-name-of-service'
Getting nginx+php-fpm to work together, but each in their own container is pretty tough. Here is a github repo where I setup everything manually without docker-compose.yml. Nginx and php-fpm are each in their own container: https://github.com/timogoosen/mediawiki-dockerized/blob/master/nginx/sites-enabled/site.conf That is the nginx config. You will see the nginx container connects to php-fpm container using a hostname and port.
fastcgi_pass php-fpm-container:9000;
See the README, it is pretty descriptive: https://github.com/timogoosen/mediawiki-dockerized/tree/master/nginx
I suggest break the problem up into small parts as you have been doing. First get nginx and php-fpm each in their own container working then serve a random page that just contains:
<?php
phpinfo();
php>
Once you get that working then go from there.
Here are some links to help you: https://stackoverflow.com/questions/39901311/docker-ubuntu-bash-ping-command-not-found https://stackoverflow.com/questions/30172605/how-to-get-into-a-docker-container https://stackoverflow.com/questions/30151436/how-to-run-nginx-docker-container-with-custom-config