Tuning Linux IP routing parameters -- secret_interval and tcp_mem

I never ever encountered this issue. However, you should probably increase your hash table width in order to reduce its depth. Using "dmesg", you'll see how many entries you currently have:

$ dmesg | grep '^IP route'
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)

You can change this value with the kernel boot command line parameter rhash_entries. First try it by hand then add it to your lilo.conf or grub.conf.

For example: kernel vmlinux rhash_entries=131072

It is possible that you have a very limited hash table because you have assigned little memory to your HAProxy VM (the route hash size is adjusted depending on total RAM).

Concerning tcp_mem, be careful. Your initial settings make me think you were running with 1 GB of RAM, 1/3 of which could be allocated to TCP sockets. Now you've allocated 367872 * 4096 bytes = 1.5 GB of RAM to TCP sockets. You should be very careful not to run out of memory. A rule of thumb is to allocate 1/3 of the memory to HAProxy and another 1/3 to the TCP stack and the last 1/3 to the rest of the system.

I suspect that your "out of socket memory" message comes from default settings in tcp_rmem and tcp_wmem. By default you have 64 kB allocated on output for each socket and 87 kB on input. This means a total of 300 kB for a proxied connection, just for socket buffers. Add to that 16 or 32 kB for HAProxy, and you see that with 1 GB of RAM you'll only support 3000 connections.

By changing the default settings of tcp_rmem and tcp_wmem (middle param), you can get a lot lower on memory. I get good results with values as low as 4096 for the write buffer, and 7300 or 16060 in tcp_rmem (5 or 11 TCP segments). You can change those settings without restarting, however they will only apply to new connections.

If you prefer not to touch your sysctls too much, the latest HAProxy, 1.4-dev8, allows you to tweak those parameters from the global configuration, and per side (client or server).

I am hoping this helps!

The Out of socket memory error is often misleading. Most of the time, on Internet facing servers, it does not indicate any problem related to running out of memory. As I explained in far greater details in a blog post, the most common reason is the number of orphan sockets. An orphan socket is a socket that isn't associated to a file descriptor. In certain circumstances, the kernel will issue the Out of socket memory error even though you're 2x or 4x away from the limit (/proc/sys/net/ipv4/tcp_max_orphans). This happens frequently in Internet-facing services and is perfectly normal. The right course of action in this case is to tune up tcp_max_orphans to be at least 4x the number of orphans you normally see with your peak traffic.

Do not listen to any advice that recommends tuning tcp_mem or tcp_rmem or tcp_wmem unless you really know what you're doing. Those giving out these advices typically don't. Their voodoo is often wrong or inappropriate for your environment and will not solve your problem. It might even make it worse.