Why am I only achieving 2.5Gbps over a 10Gbe direct connection between 2 machines?

Solution 1:

I remember I faced the same issue once. It was all related to LRO and GRO. You may want to disable large-receive-offload and generic-receive-offload on both ends and see if it makes the changes.

Run the following to temporarily disable those:

large-receive-offload

ethtool -K enp101s0 lro off

ethtool -K enp7s0 lro off

generic-receive-offload

ethtool -K enp101s0 gro off

ethtool -K enp7s0 gro off

By running the following you will actually examine the changes:

ethtool -k enp7s0 

please note that -k -K, uppercase makes changes and lowercase simply outputs the values

I always disable GRO and LRO on 10 NICs.

You can read more here: https://lwn.net/Articles/358910/

But try to disable it and see if it changes the speed, if it will, then I will give you the steps to make a permanent change in your /etc/network/interfaces.