IOAT enabled but not being used with Linux?
On our R610s with igb version 4.1.2 I do see the ioatdma module loading (and loaded with lsmod):
ioatdma: Intel(R) QuickData Technology Driver 4.00
ioatdma 0000:00:16.0: irq 80 for MSI/MSI-X
ioatdma 0000:00:16.1: irq 81 for MSI/MSI-X
ioatdma 0000:00:16.2: irq 82 for MSI/MSI-X
ioatdma 0000:00:16.3: irq 83 for MSI/MSI-X
ioatdma 0000:00:16.4: irq 84 for MSI/MSI-X
ioatdma 0000:00:16.5: irq 85 for MSI/MSI-X
ioatdma 0000:00:16.6: irq 86 for MSI/MSI-X
ioatdma 0000:00:16.7: irq 87 for MSI/MSI-X
We have a quad port Intel nic that should support it:
07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
However, it looks like it isn't being used for any network traffic:
[root@or-rtlb01 ~]# cat /sys/class/dma/dma*/bytes*
0
0
0
0
0
0
0
0
Does anyone know how to make it so ioat is used for network traffic?
Kernel is Linux or-rtlb01.ds.stackexchange.com 3.7.7-1.el6.elrepo.x86_64 #1 SMP Mon Feb 11 22:28:07 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
Solution 1:
How big are the packets you're sending through? By default the IOATDMA driver is only used on large packet sizes. You can view the current setting with:
cat /proc/sys/net/ipv4/tcp_dma_copybreak
If you want the value to be lower, you can set it using sysctl. For example, to set the packet size to 1, use:
sysctl net.ipv4.tcp_dma_copybreak=1
Solution 2:
Before starting, update complete server like BIOS including hard drive firmware.
Secondly, always use latest Intel Network Driver igb. Just compile it from the source.
BIOS enable both DCA and IOATDMA. In linux try this:
# modprobe ioatdma
# dmesg
You should see the ioatdma driver loaded.
Now, the igb driver is different to ixgbe which is 10GBps one. Since I dont use igb at all, I can tell you to try to stop network /etc/init.d/network stop and rmmod igb and modprobe igb again to see if there are any differences and post results if you can pls.