Improving SAS multipath to JBOD performance on Linux

Nice, well prepared question :)

I'm a speeds'n'feeds-man myself and I think you're on the money to be honest. I was half expecting to see your throughput lower than it is but what I think you've got there is a build-up in minor, and expected, inefficiencies. For instance it's very hard for a PCIe bus to get to 100% all the time, better to assume a low 90% overall rate. Given the jitter this will cause it will also mean that the PHYs won't be 100% 'fed' all the time so you lose a bit there to, same for the cache, disks, non-coalraced interrupts, IO scheduling etc. Basically it minor ineffeciency times minor inefficiency times...and so on, it ends up being more than the 5-10% expected inefficencies on their own. I've seen this kind of thing with HP DL servers talking to their MSA SAS boxes using W2K3 and then being NLB'ed over multiple NICs - frustrating but understandable I guess. That's my 2c anyway, sorry it's not too positive.