Why is REMOTE_ADDR only sometimes available as an Apache environment variable?

To avoid having to parse X-Forwarded-For in Varnish, I'm trying to just set a header on the SSL terminator (currently Apache) that stores the direct client IP in a header.

On our development machine, this works:

RequestHeader set X-Foo %{REMOTE_ADDR}e

However, in staging it doesn't. Specifically, the header is empty, as illustrated by both varnishlog:

13 TxHeader     b X-Foo: (null)

(On the development machine, this shows the IP address as expected.)

Similarly, logging REMOTE_ADDR shows that it only appears to be populated on the dev machine:

# Config
LogFormat "%{X-Forwarded-For}i %{REMOTE_ADDR}e" combined
CustomLog "/var/log/httpd/access_log" combined

# Log file, staging
<my ip> -

# Log file, development
<my ip> <my ip>

Since the dev machine is, well, a dev machine, it is different in a number of ways; however, I can't track down which difference is causing this. The versions of Apache are the same (2.2.22), and I don't see anything relevant in any of the standard config files or /etc/sysconfig/httpd. And the rest of the system is reasonably similar, since they're built off the same CentOS 5 base image.

I can't even tell from the Apache documentation whether REMOTE_ADDR is expected to exist or not as an environment variable, but it clearly works on one machine, whether by fluke or design, and the inconsistency is driving me mad.

Looking through the sources, REMOTE_ADDR is set only for these handlers that come with the server itself: mod_proxy_scgi, mod_ext_filter, mod_include, mod_isapi, mod_cgid, and mod_cgi (only they call ap_add_common_vars) so somehow one of these handlers is getting called before mod_headers or mod_log_config on your dev box but not on the staging box. (You may have other handlers.)

In less technical speak, from the doc you reference it says the same thing:

In addition to all environment variables set within the Apache configuration and passed from the shell, CGI scripts and SSI pages (emphasis mine) are provided with a set of environment variables containing meta-information about the request as required by the CGI specification.

If you use mod_ssl use 's' to get value of a variable (working on apache 2.4). This is an example to set X-Forwarded-For header to REMOTE_ADDR.

RequestHeader setifempty x-forwarded-for %{REMOTE_ADDR}s

More info about mod_headers.