nodeJS max header size in http.request

With nodeJS v0.10.28, is there a limit in the size/length of the header content in an http request?

Let me explain:

I need to consume rest services provided by a 3rd party provider. The data returned to me is in the header of the request, the body is mostly empty (120 or so chars). The amount of data in the header varies from a few chars to several 100kb.

var https = require('https');

var httpHeaders = {
    Authorization: 'Basic ' + new Buffer(user + ':' + psw).toString('base64'),
    accept: '*/*',
    'Content-Type': 'text/plain; charset=utf-8'
};
var options = {
    host: "www.website.com",
    port: 8080,            
    path: "/" ,   
    method: 'GET',
    headers: httpHeaders,
    rejectUnauthorized: false,
    requestCert: true,
    agent: false
};

https.request(options, function(res) {
    res.setEncoding('utf8');
    if (res.statusCode == 200) {
        var json = res.headers["someHeaderInfo"];
        callback(null,{ "result" : JSON.parse(json) });
    } else {
        callback({ "error" : res.statusCode  });                            
    }
}).on('data', function (chunk) {
    console.log('BODY: ' + chunk);
}).on('error', function(e, res) {
    console.log("  Got error: " + e.message);
    callback({ "error" : e.message });
}).end();

The code above works fine for smaller size headers but fails on the on('error', with "Parse Error" message on larger headers.

Removing the on error clause throws this exception:

Error: Parse Error
    at CleartextStream.socketOnData (http.js:1583:20)
    at CleartextStream.read [as _read] (tls.js:511:12)
    at CleartextStream.Readable.read (_stream_readable.js:320:10)
    at EncryptedStream.write [as _write] (tls.js:366:25)
    at doWrite (_stream_writable.js:226:10)
    at writeOrBuffer (_stream_writable.js:216:5)
    at EncryptedStream.Writable.write (_stream_writable.js:183:11)
    at write (_stream_readable.js:582:24)
    at flow (_stream_readable.js:591:7)
    at Socket.pipeOnReadable (_stream_readable.js:623:5)

Is there a limit on the header size, can it me changed? what solution do I have?

Thanks

Solution 1:

Use --max-http-header-size on the node command-line to accept larger headers. If you get "node: bad option: --max-http-header-size", upgrade to node v10.15.0 or newer. Relevant changelog

node --max-http-header-size 15000 client.js

Credit to @murad.

Solution 2:

The HTTP protocol parser that Node uses appears to be hard-coded with a maximum header size of 8KB. Relevant constant. Since that is a compile-time constant, you would have to use a custom-compiled version of Node to set that constant larger.

It really sounds like the service you are using has made a mistake by putting that much data in a header though. Headers are meant for metadata about the request body. If they have that much data to return, they should probably be including it in the request body.

You could explore using an alternate HTTP parser like http-parser-js, since it doesn't appear to have a limit.

Solution 3:

In a recent update to Node.js the default allowed maximum header size has recently changed from 80KB to 8KB:

The total size of HTTP headers received by Node.js now must not exceed 8192 bytes.

In our case, we updated versions of Node and all of a sudden started getting 400 Bad Request's from our express server, which was confusing to say the least.

There's an issue open to change this response to a 431 Request Header Fields Too Large which will be more descriptive and helpful.

It has also been made configurable because it was a breaking change for some:

--max-http-header-size=size#
Added in: v11.6.0
Specify the maximum size, in bytes, of HTTP headers. Defaults to 8KB.

Solution 4:

Headers should contain meta data - if they pass 80KB that means that you doing something wrong.

Node.js allows 80KB of maximum size in header data.

You can read more about http headers in here: https://nodejs.org/en/docs/guides/anatomy-of-an-http-transaction/