How to tune TCP for high-frequency connections between two nodes

Solution 1:

In our data center we have an F5 running on BigIP hardware that acts as single ingress point for HTTPS requests from client machines in our various office locations across the country.

If this single point (front-end) remains single when it passes connections down to back-end why are you wondering about the hiccups? Specially if intensity of connections is "possibly 100+ per second".

Your setup is basically squeezing one set with higher cardinality into another one with cardinality significantly lower.

ultimately only reduce the chance of those "collisions"

This is put into basis of how packet switched networks work. Say, on Ethernet level there're collisions too. Randomness is inevitable and TCP/IP is dealing with it. The IP protocol itself was built not with LANs in mind, actually (but still works great there too).

So yes "Add more source IPs and/or make Traefik listen on multiple ports" is pretty reasonable way to follow.

Solution 2:

Although I also think adding more IP addresses is the simplest way forward, have you considered exploring reusing TCP connections between the F5 and the Traefik nodes instead of creating a new one per external request?

I'm not sure how F5 supports that, but maybe it's as simple as switching to http2 between the F5 and the Traefik nodes. See https://developers.google.com/web/fundamentals/performance/http2#one_connection_per_origin