How to fine tune TCP performance on Linux with a 10Gb fiber connection

In Linux/Intel I would use following methodology for performance analysis:

Hardware:

• turbostat
  Look for C/P states for cores, frequencies, number of SMIs. [1]
• cpufreq-info
  Look for current driver, frequencies, and governor.
• atop
  Look for interrupt distribution across cores
  Look for context switches, interrupts.
• ethtool
  -S for stats, look for errors, drops, overruns, missed interrupts, etc
  -k for offloads, enable GRO/GSO, rss(/rps/rfs)/xps
  -g for ring sizes, increase
  -c for interrupt coalescing

Kernel:

• /proc/net/softirq[2] and /proc/interrupts[3]
  Again, distribution, missed, delayed interrupts, (optional) NUMA-affinity
• perf top
  Look where kernel/benchmark spends its time.
• iptables
  Look if there are rules (if any) that may affect performance.
• netstat -s, netstat -m, /proc/net/*
  Look for error counters and buffer counts
• sysctl / grub
  So much to tweak here. Try increasing hashtable sizes, playing with memory buffers, congestion control, and other knobs.

In your case your main problem is interrupt distribution across the cores, so fixing it will be your best corse of action.

PS. Do not forget that in those kinds of benchmarks kernel and driver/firmware versions play a significant role.

PPS. You probably want to install the newest ixgbe driver from Intel[4]. Do not forget to read README there and examine scripts directory. It has lots of performance-related tips.

[0] Intel also has nice docs about scaling network performance
https://www.kernel.org/doc/Documentation/networking/scaling.txt
[1] You can pin your processor to a specific C-state:
https://gist.github.com/SaveTheRbtz/f5e8d1ca7b55b6a7897b
[2] You can analyze that data with:
https://gist.github.com/SaveTheRbtz/172b2e2eb3cbd96b598d
[3] You can set affinity with:
https://gist.github.com/SaveTheRbtz/8875474
[4] https://sourceforge.net/projects/e1000/files/ixgbe%20stable/

Are the servers of the same specs (make and model)? Have you made any sysctl.conf changes?

You should enable irqbalance because your interrupts are only occurring on CPU0.

If you aren't using a tuned profile with EL6, you should choose one that's close to your workload, according to the schedule here.

Speed 6 Gb/s is ok, if you run only one instance of iperf, as it's limited to single CPU core. Two processes simultaneously should give you expected 10Gb/s.

The problem with 20Mb/s in one direction looks like driver/firmware/hardware incompatibility issue.

I suggest you to try the following troubleshooting steps:

Your NICs have dual ports, so first, try loopback speed tests on both NICs. It can help you to localize the problem: on server A or on server B. 2. Change patch cords. 3. Try new drivers. 4. Upgrade firmware. 5. Change NICs)

I would try disabling LRO ( Large Receive Offload )... I'd guess you have one with it turned on, and one with it turned off.

It's NIC / driver dependent, but in general, when we see that in our environment, we know we missed one, and go disable LRO