Max SAS throughput of a disk stack?
I've been trying to figure out something.
I have a stack of SAS shelves (attached to a NetApp if that's relevant). Each devices within the shelf is 6G SAS. According to the vendor, the sustained transfer is somewhere around 200-250MB/sec.
So - with 10 shelves of 24 drives - attached 'top and bottom' to separate controllers on my filer head.
What's the fastest rate I could transfer data from my drives?
6G SAS implies 600MB/sec. 2 controllers therefore, 1200MB (in optimal circumstances). Or - about 6 drives worth, out of my 240 spindles. This seems oddly low - am I missing something? Do the SAS controllers have some sort of mulitplicative factor?
Or am I really honestly in a position where I'll never get anywhere near 'max throughput' of the drives in this stack? Certainly it looks like my historic peak has been around 2000MB/sec on 3 controllers. (So ~650MB/sec each).
Still, I suppose it makes the drive utilisation look low....
Theoretically max throughput of NetApp stack will be 9600 MB/sec. NetApp supports 240 HDDs per stack or 96 SSDs per stack. But it's not a system limit. You can have several stacks in one system. It depends on controllers model.
SAS 6G one lane gives 600 MB/sec. But one SAS 6G port utilises 4 lanes. With NetApp you use 2 ports on each controller for one stack which in sum will give 4 ports.
So theoretical throughput = one lane rate * 4 lanes per port * 4 ports per stack = 600 MB/sec * 4 * 4 = 9600 MB/sec.
When you are talking about performance and utilisation you need to understand that there are two metrics of performance - MB/s and IOPS (actually three, latency also matters).
Different disk types have different IOPS and MB/s performance. NetApp uses these values in documentation:
SAS 10K 140 IOPS 198 MB/sec
SATA 7.2K 75 IOPS 134 MB/sec
When you look at transfer rate of disk it seems like just 48 SAS HDDs will saturate stack SAS ports. 48 * 198 = 9504 MB/sec. But majority of enterprise applications works with small block size (4KB, 8KB). And they are more sensitive to IOPS performance and latency. So one SAS 10K HDD has ≈140 IOPS. With 8KB block it will be just 8 * 140 = 1120 KB/s of throughput. 240 fully utilised disks will perform only 240 * 1120 KB/s = 262,5 MB/s throughput.
There are some high throughput workloads. For example video surveillance or streaming, data analytics. And for such workloads it may be better to have less disk shelves per stack.
What kind of workload do you have?
SAS 6 implies not 600mb.
SAS uses 4 channel per cable and a disc can be connected to 2 cables at the same time.
That gives you 4.8 gigabyte/second out of a disc.