What is wrong with my nginx reverse proxy configuration, with single server (and more later)

I'm trying to get an nginx reverse proxy setup working. I have two web servers set up, once with nginx, and one with apache2. I'm currently not able to get it working with just the nginx server, so that's all I'm trying for right now, but I added that I'm eventually trying with two, in case that effects the setup.

I have four machines in this setup.

1. Client machine

192.168.0.5

Ubuntu 14.04 desktop

2. Reverse proxy server

192.168.0.10

nginx 1.4.6

Ubuntu 14.04 server

3. Server 1

192.168.0.15

server1.mydomain.com

nginx 1.4.6

Ubuntu 14.04 server

4. Server 2

192.168.0.20

server2.mydomain.com

apache2

Ubuntu 14.04 server

On my client machine, I have set my hosts file to point to the reverse proxy server for each of the web servers, like below

/etc/hosts On client 192.168.0.5 127.0.0.1 localhost 192.168.0.10 server1.mydomain.com 192.168.0.10 server2.mydomain.com

I have ssl certs for server1 and server2, that I have put on the reverse proxy server (192.168.0.10). We'll call them server1.crt, server1.key, and server2.crt, server2.key.

I believe that I have to have this setup with the certs working like this:

client(192.168.0.5) ---https---> reverseProxy(192.168.0.10 holds ssl certs) ---http---> server1 or server2 

I have both servers working now, with http, and I just need to fix the nginx reverse proxy settings on 192.168.0.10.

Here's something I've tried, but it isn't correctly redirecting. Once again, I'd like an https connection to the reverse proxy server, and then an http connection between the reverse proxy and the servers.

/etc/nginx/nginx.conf

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
    worker_connections 768;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # nginx-naxsi config
    ##
    # Uncomment it if you installed nginx-naxsi
    ##

    #include /etc/nginx/naxsi_core.rules;

    ##
    # nginx-passenger config
    ##
    # Uncomment it if you installed nginx-passenger
    ##

    #passenger_root /usr;
    #passenger_ruby /usr/bin/ruby;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

/etc/nginx/sites-available/default

server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    # Make site accessible from http://localhost/
    server_name localhost;

    location / {
            # First attempt to serve request as file, then
            # as directory, then fall back to displaying a 404.
            try_files $uri $uri/ =404;
            # Uncomment to enable naxsi on this location
            # include /etc/nginx/naxsi.rules
    }

server {
    listen 443;
    server_name server1.mydomain.com;

    ssl on;
    ssl_certificate /usr/local/nginx/conf/server1.crt;
    ssl_certificate_key /usr/local/nginx/conf/server1.key;
    ssl_session_cache shared:SSL:10m;

    ssl_session_timeout 5m;

    ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers "HIGH:!aNULL:!MD5 or HIGH:!aNULL:!MD5:!3DES";
    ssl_prefer_server_ciphers on;

    location / {
            proxy_pass http://192.168.0.15:80;
            proxy_set_header Host server1;

            proxy_redirect http:// https://;
    }
}

I'm assuming that something is incorrect in my /etc/nginx/sites-available/default file, but I've been reading through several tutorials, and this seems pretty close. This setup, is obviously only trying with server1, and ignores server2, but I assumed that if I could get one work, I could add as many others as I wanted. I have found similar questions, such as this one, but I still haven't been able to get this configuration working with a single server.

What's currently happening

Currently, when I go to server1.mydomain.com, I get the standard "Welcome to nginx!" page from the reverse proxy server (192.168.0.10). There's no forwarding going on.

Am I getting close? Thanks in advance

EDIT1

After trying the solution posted by Capile, I ran into another problem (which may have been expected, by someone with more web knowledge than myself).

When I changed my /etc/nginx/sites-available/default file to this:

/etc/nginx/sites-available/default (On reverse proxy)

server {

        listen 80 default_server;
        listen 443 ssl default_server;
        server_name server1.mydomain.com;

        ssl on;
        ssl_certificate /usr/local/nginx/conf/server1.com.crt;
        ssl_certificate_key /usr/local/nginx/conf/server1.com.key;
        ssl_session_cache shared:SSL:10m;

        ssl_session_timeout 5m;

        ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers "HIGH:!aNULL:!MD5 or HIGH:!aNULL:!MD5:!3DES";
        ssl_prefer_server_ciphers on;

        location / {
                proxy_pass http://192.168.0.15:80;
                proxy_set_header Host server1;

                proxy_redirect http:// https://;
        }
}

With this config I get a

400 Bad Request

The plain HTTP request was sent to HTTPS port

I thought maybe I should try https://server1.mydomain.com, but that just spins.

Also, I don't mind using the same ssl cert for both servers. I don't think that will be an issue.

EDIT2

First of all, thank you for all of the help.

I removed the ssl on; line as recommended by Capile, and changed the proxy_redirect http:// https://; line to proxy_redirect http:// $scheme://; as recommended by Richard Smith.

This fixed the bad request for the http traffic. So now, if I go to http://server1.mydomain.com I am successfully redirected to the site (yaay!)

If I try to go to https://server1.mydomain.com I am also redirected, which is great, but I'm getting an Unable to connect error. This makes sense if the reverse proxy is forwarding http traffic to http, and https traffic to https since there is no https configuration for the backend server.

My goal is that if I go to http://server1.domain.com that it connects to the reverse proxy using https, and then it forwarded on to the backend server using http. That doesn't appear to be happening...it looks like it's just forwarding it without ever using https.

On the other hand, if I go to https://server1.domain.com it should connect to the reverse proxy using https, and then forward on to the backend server using http.

So I never want an http connection from the client to the reverse proxy server.

The curls are acting as expected though. When I curl the http, or https site, I get this:

curl -i https://server1.mydomain.com or curl -i http://server1.mydomain.com

HTTP/1.1 302 Found
Server: nginx/1.4.6 (Ubuntu)
Date: Thu, 21 Jan 2016 16:34:29 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 92
Connection: keep-alive
Cache-Control: no-cache
Location: http://test1/users/sign_in
Set-Cookie: session=336e109ad711; path=/; expires=Thu, 28 Jan 2016 16:34:36 -0000; HttpOnly
Status: 302 Found
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: ad65f-f6ds204-9fs8d-bdsdfa43-df5583266sdf87
X-Runtime: 0.005358
X-Xss-Protection: 1; mode=block

<html><body>You are being <a href="http://test1/users/sign_in">redirected</a>.</body></html>

So, the redirect is definitely happening for either type of connection, but I don't THINK that the SSL certs for the https connection between the client and reverse proxy server are ever being used.

Solution 1:

you are close, but there's no reverse proxy for http configured, only for https (in the above setup) — so it should show default content from document root. First server also lacks ending }.

You may configure both http and https in the same block, just use the ssl keyword in the listen directive (under the ssl port, so there's no need to set the ssl on directive):

server {
    listen 80 default_server;
    listen 443 ssl default_server;
    ...
    ssl_certificate /usr/local/nginx/conf/server1.crt;
    ssl_certificate_key /usr/local/nginx/conf/server1.key;
    ssl_session_cache shared:SSL:10m;
...

I must alert that using two different SSL certificates within the same IP is somewhat limited and complicated — that is because it would require HTTP renegotiation (since the request needs to be encrypted with the certificate before requesting the proper Host), so you'll probably need to separate the certificate by server block. For this, use a more specific, by IP listen:

server {
    listen 192.168.0.10:443 ssl; # for server1.mydomain.com
...

server {
    listen 192.168.0.11:443 ssl; # for server2.mydomain.com
...

I also recommend you use the Mozilla SSL Config Generator for best practices on SSL security.

EDIT1

To avoid the 400 error, just remove the directive ssl on; and reload. It was causing the http traffic to require SSL — you already indicated to use ssl on the line: listen 443ssldefault_server; (this means turn ssl on on this port).

The rest of the configuration is fine, but depending on your server response, you may need to adjust the proxy settings.

First, check if the spinning is client-side or server-side: access it with cURL or turn on web developer tools on your browser and check the network: if your browser is being redirected, you probably need to adjust the proxy_redirect or even make some string replacements on your server response.

Check it this way with cURL:

curl -i http://server1.mydomain.com

If you see a Location: header in your response, you'll need to finetune your proxy settings (that will depend on the response). For example, you may be accessing it as https then your proxy forwards as http and your backend server application redirects to https (but when the response goes through the proxy, it goes back to http to the browser). There are several ways of fixing it, by using proxy_set_header or even adjusting your backend server.

For example you could use:

proxy_set_header X-Forwarded-Proto $scheme;

But either your backend http server or your application would need to properly understand this.

Alternatively, if there's no redirect, but no response as well (or a No gateway response after the request timeout — 30s), check if your backend server is properly responding to the proxy server.

Please also note that you don't need a http server on your backend, you can use, for example, a FastCGI server directly — this sometimes has fewer downfalls.

EDIT2

Based on the cURL response, I see that the configuration is working as expected — the backend server response is asking for you to sign in. If the SSL certs weren't working, you wouldn't be able to curl -i https://server1.mydomain.com.

The traffic between frontend (proxy) and backend are made through http only (see the proxy_pass directive), and that's also usually expected (another encryption might add unnecessary overhead).

Now, if you wish to use https only, you have two options: either configure that in your backend (so that it forwards you to https://test1/users/sign_in) or use a different setting for nginx, where you strip the http:80 server and make it redirect everything to https:443. Something like this (be sure to remove the listen 80 from the next server block):

server {
    listen 80 default;
    server_name  _;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
... 

Solution 2:

With you new (EDIT1) config, any access to machine #2 using http (as opposed to https) will be proxied. Which eliminates the original "Welcome to nginx!" side-effect, which was (probably) caused by a redirect to http scheme.

Moving on, I think the root of the problem is machine #3 (the real server 1) generating a redirect which is not being mapped correctly by the time it gets back to the client.

You might want to check the access logs on machine #2 and machine #3 to fully understand the interaction. You may need to adjust the log format to understand which scheme is used (http vs https) for each interaction.

You might want to temporarily disable proxy_redirect so that you can use developer tools at the client to understand exactly which URLs machine #3 is emitting in its http 3xx responses.

Your proxy_redirect directive is wrong for the EDIT1 configuration as you are now proxying both schemes, so this might be more appropriate:

proxy_redirect http:// $scheme://

Having said that, a more specific proxy_redirect may be appropriate. As the documentation states, you can use multiple proxy_redirect rules. I am not an expert onproxy_redirect` rules, but you may end up with something like this:

proxy_pass http://192.168.0.15;
proxy_set_header Host server1;
proxy_redirect http://192.168.0.15 $scheme://
proxy_redirect http://server1 $scheme://