Securing access to certain pages/directories with Apache behind a Varnish cache?

I have a public website with some URLs/directories which are for private/internal use only. These private areas can only be accessed via certain IP addresses or with a known username/password.

Currently I achieve this via .htaccess files like this:

AuthType     Basic
AuthName     "Protected Area"
AuthUserFile /path/to/.htpasswd

SetEnvIf Remote_Addr     1.2.3.4 trusted
SetEnvIf X-Forwarded-For 1.2.3.4 trusted
# (Note I am aware X-Forwarded-For can be spoofed)

<RequireAny>
    Require env trusted
    Require valid-user
</RequireAny>

The problem is I want to add Varnish in front of my server to provide caching. Obviously the existing setup won't work with Varnish because it can't cache content restricted by .htaccess files in this way.

Is there a way I can continue to use my .htaccess files to protect my internal pages, or is there a similar approach I can use to place the responsibility for security on Varnish itself which doesn't require changing Varnish's VCL files every time I want to add or modify the restrictions?

You can definitely do this in Varnish by writing the following VCL:

vcl 4.0;

acl allowed {
  "1.2.3.4";
}

sub vcl_recv {
    if(!client.ip ~ allowed && req.http.Authorization != "YWRtaW46YWRtaW4K") {
        return (synth(401, "Restricted"));
    }
    return (hash);
}

sub vcl_synth {
    if (resp.status == 401) {
        set resp.http.WWW-Authenticate =
            {"Basic realm="Restricted area""};
    }
}

Limitations

Although this works fine, the username/password checks are done manually. If you have multiple usernames/passwords, you'll have to add each user in the if-statement.

Make sure you bypass the default VCL behavior

When handling Authorization headers in Varnish, it's important not to fall back on default behavior. By default Varnish will not lookup items in cache when an Authorization header is present.

So at some point in your logic, you'll have to execute return (hash);, which I did in the example. Otherwise you'll fall back on default behavior and items will not be looked up in cache.

Vmod_basicauth as an alternative

If that turns out to be a deal breaker, you can compile the following Varnish module that allows you to load an .htpasswd file and perform the authentication for you.

Still doing it all in Apache

If you decide to keep doing this in Apache, you'll need to change the VCL as well. I explained that default behavior will stop Varnish from looking up items in cache when an Authorization header is present.

If you again look at the default behavior again, you'll have to make sure a return (hash); is executed before you hit default behavior, or you have to reimplement part of it your self by adding the necessary if-statements