How can 8 SAS lanes support 1024 disks? [duplicate]

Solution 1:

How can 8 SAS lanes support 1024 disks?

Using SAS expanders - each SAS channel can theoretically support 65,536 devices per link using expanders - 8 channels can easily support 1024 disks, though it would be horribly contended with 128 disks per channel.

If I bought the 12Gb/s LSI JBOD version, would I then get better performance when the disks are still 6Gb/s?

Each channel will run at 6Gbps because that's the speed of the slowest device on the channel, so no, 12Gbps channels won't help in this case.

Out of interest how come you chose to use that controller and not the supported P888 controller from HP?

Solution 2:

This is due to SAS support for expanders.

Basically, an expander is akin to a port multiplier / network switch, where a single upstream SAS connection can be expanded in up to 255 different downstream SAS link. As your HBA/RAID card supports 8 upstream links each with the capability to be expanded to 255 downstream links, the total SAS capacity is at 1024 SAS devices.

Bandwidth is another matter, of course: while you can connect up to 1024 SAS device, total disk bandwidth from the controller to the disks (and vice versa) can not exceed the total bandwidth of the physical 8 SAS connections. In other words: you can be link-bandwidth starved in some scenarios.

For more informations about SAS expanders you can see this Wikipedia article.

Speaking about 6Gb/s vs 12Gb/s, with the right card and in the right condition (read: many drives accessed by the means of an expander card) you can have a performance advantage even when using 6Gb/s. Basically, even if your disks peak at 6Gb/s, your HBA/RAID adapter can communicate with the expander card at 12Gb/s. For example, consider this Adaptec card:

What are the performance benefits? The Adaptec SAS expander card features SAS and SATA edge buffering capabilities that enable it to increase total throughput to the controller even when using 6Gb/s SAS drives. This is achieved by bursting data at 12Gb/s when connected to 6Gb/s SAS or SATA devices when utilized in conjunction with a 12Gb/s capable adapter such as the Adaptec Series 8 SAS RAID or SAS HBAs.