Disk size addressable by the OS

I have a 3ware 9550SXU-12 storage controller and 750G disks connected to it. The disks are configured as single units (not JBOD).

I've been running some performance tests, mainly to see the impact of partition alignment, encryption, raid level, etc. on read/write/iops performance.

I was surprised that in my case the read/write performance for the same storage configuration was slightly lower with aligned partition than with unaligned ones.

This prompted me to start checking if there's a difference in how the disk is visible to the OS when connected via the 3ware controller and when using the port on the motherboard, which doesn't come with any RAID support at all.

I know about the Disk Control Block (DCB) metadata that 3ware controllers put on the disks to allow for replacing the controller without having to reconfigure it as the configuration data is read from the DCB block on the disks. My controller uses the 'New Format' which apparently means that the controller writes the DCB in the last 1024 LBAs of the disk.

I was interested to see if my alignment efforts weren't being foiled by the 3ware controller presenting only part of the disk to the OS.

What I've found:

• the beginning of the disk looks exactly the same when connected with or without 3ware controller. There shouldn't be any impact on the alignment here. Verified with dd/md5sum.
• the last 1024*512B of the disk indeed contains something that looks to be put there by 3ware (judging by the readable strings there)
• now the interesting part: when under 3ware control, the disk reports media size of 749988741120 B and when connected directly - 750156374016 B, which means that when connected through the 3ware controller, the OS has access to about 160MB less of disk media.

It would be understandable if it was just a difference of 1024x512B (the DCB) but 160MB seems a bit too much space to store this type of controller metadata.

QUESTIONS:

Does anyone know if there are other considerations when aligning paritions on disks connected to controllers which store unit configuration on those disks, that I might have missed?

Out of curiosity - does anyone know what the last 160MB of disk media is used for?

Thanks

I can't comment directly, as I'm not familiar with 3ware.

However in general, I've run into quite a few storage arrays that 'steal' some disk space. There's a variety of reasons including:

• The controller runs a cut down OS, and needs space. (Config/management etc)
• The controller runs a write cache, and in the event of a power failure needs somewhere to emergency flush the cache.
• Size normalization, to allow slight variance in disk sizes.
• a built in 'allocation unit' in the controller, e.g. only allowing allocations of fixed multiples. (Usually geared to cache/paging limits).

Regarding alignment - the only reason I can think of for your correct alignment to be slower, is if your controller is also handling the alignment. Increasingly arrays/controllers are host OS aware - partly because of needing to correctly set SCSI flags, but also because of this alignment problem.

You may find that if your array knows the platform of your host, then it's 'adjusted' the alignment internally already. (and thus by aligning yourself, you've misaligned it again).