Why am I not receiving a response when the request is sent through a load balancer and the response is too big to fit into a single packet?

There's a certain condition where I'm unable to receive a response from an HTTP request to a particular service: the request is sent through a load balancer and the response is too big to fit into a single packet. Under those conditions, a response is never received. Neither is an error. The server sends a TCP ACK in response to the packet containing the HTTP request, but nothing comes after that. The connection just hangs waiting for something to happen. The behavior is observed whether the request is sent via CURL, node's HTTP library, or Postman.

What could cause that behavior? How can I debug what is causing the issue?

• I have verified that the service behind the load balancer is receiving and responding to the HTTP request, but the response is getting lost somewhere.
• We have other services with a similar setup that don't exhibit this issue.
• We tried setting a different instance of the load balancer but it had the same problem.

In tshark on the server, I see a handful of [TCP Retransmission] entries after it receives one of the problematic requests through the load balancer (this doesn't happen when the same request is received directly). This seems to indicate that something between the client and the server is misdirecting TCP traffic (highly implicates the load balancer) but I can't understand what would cause that to occur only when HTTP responses are split into multiple packets, and only by this particular service.

Conditions that cause the failure:

Network topology:

Client -> Load Balancer -> Service

Solution 1:

It turns out that I discovered a PMTUD black hole.

I first suspected a fragmentation problem due to an article from Cloudflare about packet fragmentation. I confirmed the issue using ping to send different packet sizes: ping -s SIZE.

First I tried 1500 which is the default MTU for ethernet.

> ping 172.16.5.1 -s 1500

But nothing happened. It just hung and printed nothing. If there's a problem, you're supposed to get an error like this:

ping: sendto: Message too long
Request timeout for icmp_seq 0

Eventually, I found an article that described the issue in more detail and recommended a fix: setting /proc/sys/net/ipv4/tcp_mtu_probing to 1 to enable black hole discovery (which is not enabled by default in Linux 🤷‍♂️). I wasn't actually able to modify that file, so I followed the recommendation to set it in /etc/sysctl.conf and then reload settings via sysctl --system. After all that, I was able to receive responses from the server.

MTU probing adds overhead to the response (it was noticeably delayed) so I still intend to figure out why ICMP packets are not being received, but at least I now understand the problem and have a workable solution.

Update: Turns out a LAG had an MTU of 1500 so it was dropping the packets.