Access Proxy IP in <If> directive with mod_remoteip
I should have read the documentation about expression variables more carefully, as it says there:
[...] | REMOTE_ADDR | The IP address of the remote host [...] | CONN_REMOTE_ADDR | The peer IP address of the connection (see the mod_remoteip module) [...]
So there is a predefined variable CONN_REMOTE_ADDR, and
<If "%{CONN_REMOTE_ADDR} != 'my.proxy.ip'">
RequestHeader unset Trusted-User
</If>
does the trick.