How to log all headers in nginx?
How do I log all the headers the client (browser) has sent in Nginx? I also want to log the response headers. Note that I am using nginx as reverse proxy.
After going through documentation, I understand that I can log a specific header, but I want to log all of the headers.
Solution 1:
After much research, I can conclude that it is not possible out of the box.
Update- you can use openresty which comes with Lua. Using Lua one can do pretty cool things, including logging all of the headers to say, Redis or some other server
Solution 2:
As @gauravphoenix said you need openresty which comes with Lua. See https://github.com/openresty/lua-nginx-module/ for installing it. Once it's running then add in nginx
header_filter_by_lua_block {
local h = ngx.req.get_headers()
for k, v in pairs(h) do
ngx.log(ngx.ERR, "Got header "..k..": "..toString(v)..";")
end
}
Inspect your error log.
Solution 3:
Based on @user1778602’s response the set_by_lua_block can be used to set all headers into a variable to be consumed later at the log_format (see more details in this answer).
set_by_lua_block $request_headers{
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
local rowtext = ""
rowtext = string.format("[%s %s]\n", k, v)
request_headers_all = request_headers_all .. rowtext
end
return request_headers_all
}
The update your log_format (be sure to use the new format in e.g. access_log):
log_format myformat escape=none "$request_headers"
PS: Beware of logging PII data may be violating GDPR :)
Solution 4:
There are two more options to log headers in nginx.
- nginx njs scripting language can be used instead of lua (njs may be considered easier to install and to use and is somewhat more "official")
- mirror directive (only for request headers)
Using njs to log request and response headers
njs can be installed from package repositories and is preinstalled in official nginx docker images. njs is available since at least 1.9.15 version of nginx (which is rather old), but it is better to use more recent versions.
After installation enable njs http module in nginx config:
load_module modules/ngx_http_js_module.so;
Headers may be logged to error or access log.
Logging to access log with njs
- Decide which format to use (JSON, custom, base64...)
- Create js module with function to convert headers structure to string (~3 lines of code)
- Import this js module in nginx configuration (~1 line)
- Declare a variable to use in log_format directive (~1 line)
- Add this variable to log format (~1 line)
HTTP Request object has headersIn, headersOut fields in key value format, duplicate headers are merged in this fields and rawHeadersIn, rawHeadersOut which are array of arrays of raw headers.
Create js module, use json to serialize headers:
# /etc/nginx/headers.js
function headers_json(r) {
return JSON.stringify(r.headersIn)
}
export default {headers_json};
Import js module, declare variable and add it to log_format:
http {
js_import headers.js;
js_set $headers_json headers.headers_json;
# Using custom log format here
log_format main '$remote_addr'
'\t$remote_user'
'\t$time_local'
'\t$request'
'\t$status'
'\t$headers_json';
Escaping in acccess log
By default strings in access log are escaped, so you will get something like this:
# curl -H 'H: First' -H 'H: Second' localhost:8899
172.17.0.1 - 16/Apr/2021:08:46:43 +0000 GET / HTTP/1.1 200 {\x22Host\x22:\x22localhost:8899\x22,\x22User-Agent\x22:\x22curl/7.64.1\x22,\x22Accept\x22:\x22*/*\x22,\x22H\x22:\x22First,Second\x22}
You can use escape parameter in log_format directive to change how escaping is applied.
Example of escape=json output:
log_format main escape=json ...
{\"Host\":\"localhost:8899\",\"User-Agent\":\"curl/7.64.1\",\"Accept\":\"*/*\",\"H\":\"First,Second\"}
Another option is to wrap json in base64 encoding in javascript function:
function headers_json_base64(r) {
return JSON.stringify(r.headersIn).toString('base64')
}
Logging to error log with njs
With njs you can use ngx.log or r.log (for older versions of njs ngx object is not available) in javascript function to log headers. js function should be called explicitly for this to work for example with
js_header_filter directive.
js module:
function headers_json_log(r) {
return ngx.log(ngx.WARN, JSON.stringify(r.headersIn))
}
export default {headers_json_log};
Enable logging in location:
location /log/ {
js_header_filter headers.headers_json_log;
return 200;
}
For error log escaping is not applied, so you will get raw json:
2021/04/16 12:22:53 [warn] 24#24: *1 js: {"Host":"localhost:8899","User-Agent":"curl/7.64.1","Accept":"*/*","H":"First,Second"}
Logging to error log may be useful if you don't want to mess up your access logs or you need to log headers only for specific location (for specific location access_log directive and separate log_format can be used too)
Logging request headers with mirror directive
mirror directive can be used to mirror requests to different location. It may be more convenient than tcpdump especially when upstream traffic is encrypted and is a bit simpler than using njs.
mirror can be used only to capture request headers since response headers are returned independently.
mirror directive can be used server wide in http and server contexts or in location context.
# using mirror in server context
mirror /mirror;
mirror_request_body off;
location /mirror {
# Use internal so that location is not available for direct requests
internal;
# Use small timeout not to wait for replies (this is not necessary)
proxy_read_timeout 1;
# Pass headers to logging server
proxy_pass http://127.0.0.1:6677;
# send original request uri in special header
proxy_set_header X-Original-URI $request_uri;
}
To log headers simple http server or just netcat oneliner may be used:
nc -kl 6677 > /path/to/headers.log
Because netcat doesn't reply to nginx, nginx will fill up error log with timeout errors, this errors do not affect clients.