How to improve Intel X520-DA2 10Gb NIC throughput without Jumbo packets

One of the problems with high performance NIC's is that the modern PC architecture has a bit of trouble keeping up. But, in your case, this isn't so much the problem. Let me explain.

The CPU has to do a lot of work processing TCP packets. This affects the throughput. What's limiting things in your case is not the network hardware, but the ability of the server to saturate the network links.

In more recent times, we've seen processing move from the CPU to the NIC like checksum offload. Intel have also added features to help reduce the load further. That's cool and I'm sure all optimizing features are turned on.

As you've alluded to, jumbo frames - actually that helps throughput somewhat. But not as much as RDMA.

Most 10GBit ethernet hardware will have a very nice underutilized feature called RDMA or remote direct memory access. It allows the NIC to do memory to memory copies over the network without the intervention of the CPU. Well, OK the CPU tells the NIC what to do and then the NIC does the rest. The trouble is, it's not used much yet. But it's getting there. Apparently, in the most recent version of Microsoft Windows Server 2012, they have something called SMB Direct. It uses RDMA. So, if you want to increase throughput, you want to use that.

Are you able to put together some test hardware and install it onto there to see how it performs?

By the way, I'm not sure if you will see it at 10Gbit so much, but fast RAM helps with RDMA especially with 56Gbit Infiniband. In general, it's best to use the fastest RAM your server supports.

Also note this comment on the SMB Direct link I put above:

You should not team RDMA-capable network adapters if you intend to use the RDMA capability of the network adapters. When teamed, the network adapters will not support RDMA.

Update: Looks like not ALL 10GBit NIC's support RDMA for some reason. So check your model's features first.

Another thought I had was the type of protocol being used to do your testing may be affecting the results. i.e. protocol overhead on top of TCP overhead. I suggest you look into using something that can test without touching the hard drive such as iperf. There is a windows port of it somewhere.

I think this question: Why does my gigabit bond not deliver at least 150 MB/s throughput? is related to your problem. I was talking about a Dell PowerEdge 6950 there. The answer was basically "use jumbo frames" to reduce the interrupts. I can imagine that tuning the offload-engine of the network-card might help in your case, but I do not know how to do so on W2K8R2.

Idea: Raise number of buffers in the network card, raise the interrupt-triggor for packets in the buffer, so each interrupt will handle more packets (i.e pass them to the OS-IP-stack).

See this link: Setting coalescence-parameters with ethtool for 10 Gb this is what I am basically referring to.