How to invalidate nginx reverse proxy cache in front of other nginx servers?

I'm running a Proxmox server on a single IP address, that will dispatch HTTP requests to containers depending on the requested host.

I am using nginx on the Proxmox side to listen to HTTP requests and I am using the proxy_pass directive in my different server blocks to dispatch requests according to the server_name.

My containers run on Ubuntu and are also running a nginx instance.

I'm having troubles with caching on a particular website that is fully static: nginx keeps on serving me stale content after files updates, until I:

Clear /var/cache/nginx/ and restart nginx
or set proxy_cache off for this server and reload the config

Here's the detail of my configuration:

On the server (proxmox):

/etc/nginx/nginx.conf:

user www-data;
worker_processes 8;
pid /var/run/nginx.pid;

events {
    worker_connections 768;
    # multi_accept on;
    use epoll;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    #tcp_nopush on;
    tcp_nodelay on;
    #keepalive_timeout 65;
    types_hash_max_size 2048;
    server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    client_body_buffer_size 1k;
    client_max_body_size    8m;
    large_client_header_buffers 1 1K;
    ignore_invalid_headers on;

    client_body_timeout 5;
    client_header_timeout 5;
    keepalive_timeout 5 5;
    send_timeout 5;
    server_name_in_redirect off;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";

    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    # gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;


    limit_conn_zone $binary_remote_addr zone=gulag:1m;
    limit_conn gulag 50;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

/etc/nginx/conf.d/proxy.conf:

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_hide_header       X-Powered-By;
proxy_intercept_errors  on;
proxy_buffering         on;

proxy_cache_key         "$scheme://$host$request_uri";
proxy_cache_path        /var/cache/nginx levels=1:2 keys_zone=cache:10m inactive=7d max_size=700m;

/etc/nginx/sites-available/my-domain.conf:

server {
    listen 80;
    server_name .my-domain.com;

    access_log off;

    location / {
            proxy_pass http://my-domain.local:80/;
            proxy_cache cache;
            proxy_cache_valid 12h;
            expires 30d;
            proxy_cache_use_stale error timeout invalid_header updating;
    }
}

On the container (my-domain.local):

nginx.conf: (everything is inside the main config file -- it's been done quickly...)

user  www-data;
worker_processes  1;

error_log  logs/error.log;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    gzip  off;

    server {
        listen       80;
        server_name  .my-domain.com;
        root /var/www;

        access_log  logs/host.access.log;
    }
}

I've read many blog posts and answers before resolving to posting my own questions... most answers I can see suggest setting sendfile off; but that didn't work for me. I have tried many other things, double checked my settings and all seems fine.

So I'm wondering whether I am not expecting nginx's cache to do something it's not meant to...?

Basically, I thought that if one of my static files in my container was updated, the cache in my reverse proxy would be invalidated and my browser would get the new version of the file when it requests it...

But I now have the sentiment I misunderstood many things.

Of all things, I now wonder how nginx on the server can know about a file in the container has changed? I have seen a directive proxy_header_pass (or something alike), should I use this to let the nginx instance from the container somehow inform the one in Proxmox about updated files?

Is this expectation just a dream, or can I do it with nginx on my current architecture?

Solution 1:

What you want to do is not possible. When a URL is cached by the proxy, no further queries will be made to the backend for this URL until the expiration of the cache. There is no way for the proxy to be aware that the file has been updated on the backend.

Some advanced caches like Varnish are able to handle invalidations demand through headers or PURGE requests. For example when a user edits a mediawiki page, the response to the POST request contains a header that invalidate the cache entry for the article. But this process only works if a resource is modified using a URL call: changing a static file on the backend does not inform the proxy cache.

One can implement this type of invalidation on nginx through the lua module : http://syshero.org/post/68479556365/nginx-passive-cache-invalidation.

Solution 2:

You can achieve this functionality by using a standalone cache (e.g. memcached or redis). nginx has modules to work with them. You will need to some convention on cache keys (e.g. construct the key from user cookie and url). In this case, you can control the cache items from the backend, including invalidating and updating.

How to invalidate nginx reverse proxy cache in front of other nginx servers?

Solution 1:

Solution 2:

Related

Recent Posts