High %soft CPU usage during high network load on ixgbe

We have an Intel 82598EB 10GBE NIC. The network load is composed of lot of small sized UDP packets. To simulate a DDoS, from another server we pushed ~100k pps small UDP packet (just some Mbit of bandwidth). When we do, %soft CPU usage goes to 100% making the server unusable and dropping packets.

Kernel is 3.8.5, ixgbe 3.13.10 We have already made some tunings to lower interrupts rate (rx-usecs: 300) but didn't solve the problem. Is there a way to check why softirqs use so much CPU even if interrupts rate is not high?

Thank you!

Solution 1:

I know this is old; but I'm mainly replying for users who come here with similar issues.

There are a few ways to check interrupts:

• cat /proc/interrupts records interrupts per CPU and IO device.
• grep cpu /proc/stat column 6 and 7 contain irq and softirq stats.
• top and htop; observe si and if you change htop to advanced display mode for CPUs you can see si and hi.
• To get historical data install sysstat; to get specific interrupt data see sar -I XALL.
• You can also use perf.

Since this is most likely a driver issue (or the NIC is failing) I'd check for any overruns or errors on the interface; if there are none see if there is an update to the driver.