Why is ksoftirqd using 100% of the CPU?

Running FC release 12. Im alaways seeing ksoftirqd/x (x being 0-9) at the top of the processlist, with 100% cpu.

The server has a bonded 2gbit connection, serving files from an SSD array. Currently its using 1.6gbit. Server load is ~ 1.5 (dual quad core). iowait is non-existent.

This has been a longstanding linux problem. If you do a google search, the reports of this go way back to 2004, without any real answers. This seems to be mostly happening during high network activity, and looks like this is your case. Linux tcp stack has never been the greatest, so I am not surprised. This may also be related to your network interfaces and the modules they are using.

I would suggest trying different network hardware. Maybe shell out for a 10Gbit card if 1Gbit is not enough? Try without bonding?

Look at making sure that your nic supports MSI-X (lspci -vv).

You can read more here http://www.alexonlinux.com/msi-x-the-right-way-to-spread-interrupt-load