bonding-rr vs single 10G for NFS NAS - pros and cons?

Solution 1:

You definitely should not consider balance-rr as a bonding mode for anything other than direct point-to-point links for a single protocol to go over. The primary issue with balance-rr is that you'll get excessive jitter and out-of-order delivery, which is pretty damning for services like NFS and CIFS. Even with a point-to-point topology and one protocol, bonding the maximum supported amount of 1G links in balance-rr won't be anywhere near as reliable as a single 10G link. balance-rr is a desperate last-resort kludge for more bandwidth in a limited system.

LACP will balance connections well, but is only effective for multiple endpoints. This isn't an issue for most file and block servers, since those are typically deployed for the purpose of providing storage to multiple endpoints. Most switches max out at 8 port LACP bonds, and balancing traffic over 8 ports is far from perfect.

10G Ethernet is fairly expensive to transition to. Have you looked at 10G or 40G infiniband? It's worlds cheaper, you can run all your hosts along a homogeneous redundant fabric, and you get built in RDMA.