SAS Expander backplane performance drops when more than 8 drives are active?

This is how SAS expanders work. You're technically oversubscribed, but in reality, you shouldn't be.

• Your WD2003FYYS disks link at 3.0Gbps, but are only capable of 1.2Gbps/disk.
• Your Supermicro drive backplane/expander has 16 ports with 6.0Gbps downlinks to the drives, but a single SFF-8087 uplink to the controller.
• From what I can tell, this backplane only supports a single SAS SFF-8087 connection to an HBA. The 836EL2 model supports multiple connections.

The SAS SFF-8087 connection is capable of 4 x 6.0Gbps == 24Gbps throughput.

140 Megbytes/second is ~1.2Gbps. 8 disks at that rate is about 9Gbps. You can probably get a few more disks going before rates drop. Account for some overhead.

You're topping out at 3.0Gbps speeds (12Gbps ceiling on a 4-lane SAS cable).

There may be a link negotiation issue between your LSI HBA and the expander backplane. Check the LSI controller and see if it is showing 3.0Gbps link on the SFF-8087 port or 6.0Gbps. You're only getting 3.0Gbps speeds, but all of the devices in the chain (except for the disks) are 6.0Gbps.

Have you brought the firmware of the expander up-to-date?

Edit:

I think I ran into this at work... There was an issue with this expander type and LSI controllers that caused SATA drives to link at 1.5Gbps. Take a look here. So the negotiation issues may be on the drive side. Can you also check firmware on your LSI controller.

SAS expanders do not (as of this writing) "store and forward" like an Ethernet switch would. They can only go as fast as the slowest signaling device in the array.

In this case, your drives only support 3.0 Gbps, and the single SFF-8087 cable carries 4 ports to the controller, for a total of 12.0 Gbps. With encoding overhead, that leaves you with roughly 1200 MB/s to share between the 16 drives, roughly 75 MB/s per drive. Throw in some SATA protocol overhead and we're down to the 60 MB/s you are seeing.

If you add another SFF-8087 cable for a total of 8x 3.0 Gbps channels , you will double the available bus bandwidth, realistically achieving ~120 MB/s per drive. Conversely, if the drives supported 6.0 Gbps instead of 3.0, you could achieve that same 120 MB/s over a single cable, or hit each individual drive's peak throughput over two cables. (TL;DR: use two SFF-8087 uplinks whenever possible)

This is one of the reasons why SATA and SAS drives now ship with 6.0 or even 12.0 Gbps signaling rates. The individual disks can only deliver a small fraction of that throughput, but by running the entire SAS bus at a faster rate, it can handle more drives before bottlenecking.