How to limit TCP connections with nginx for HTTP/2?

I want to use nginx as a https frontend and I'm looking for a way to limit TCP connections. These limits should be based on the remote IP addresses and work for HTTP/1.1 as well as HTTP 2.

For HTTP/1.1 I thought I would be fine around 15 concurrent connections per IP (since most browsers seems to use at most 6), wich should be more than enough for HTTP/2. But the nginx docs states:

In HTTP/2 and SPDY, each concurrent request is considered a separate connection.

And the HTTP/2 spec states about the SETTINGS_MAX_CONCURRENT_STREAMS parameter:

It is recommended that this value be no smaller than 100, so as to not unnecessarily limit parallelism.

Does this mean that I should allow at least 100 connections per remote IP? It seems so in my tests, but maybe I'm missing something. Because this seams to essentially mean that I can't set any meaningful limits for the number of concurrent TCP connections, without seriously limiting HTTP/2.

Here's how I solved it for now:

map $http2 $v1ip {
    default "";
    ""  $binary_remote_addr;
}
map $http2 $v2ip {
    default $binary_remote_addr;
    "" "";
}

limit_conn_zone $v1ip zone=v1ips:10m;
limit_conn v1ips 10;

limit_conn_zone $v2ip zone=v2ips:10m;
limit_conn v2ips 125;

Since the connection limit for HTTP/2 has to be that high, I also added rate limiting… but I'm still tweaking the values.

NOTE: while this allows setting a "sensible" value for HTTP < 2 connections, it still requires an — imo unnecessary — high number of concurrent connections for HTTP/2 connections. I would very much like to see limit_conn to only care for TCP connections and not HTTP requests.