New RAID hdparm slow

I just got a HP DL180 G6 that has 25X 146GB 15K SAS drives, 36GB RAM, 2X 2.0GHz Xeon 1333Mhz FSB. For fun I configured them all in a single RAID 0 and installed Ubuntu on it to see how fast it could get with 25 drives on a HP Smart Array P410 Controller w/ 512MB RAM.

When I ran hdparm -tT /dev/mapper/concorde--vg-root I get

Timing cached reads:   5658MB in  1.99 seconds = 2834.13 MB/sec
Timing buffered disk reads: 1192 MB in  3.00 seconds = 397.13 MB/sec

When I run the same command on my other server (HP DL360 G5 - 32GB RAM - 2X 2.66GHz 667Mhz FSB) that only has 4X 15K drives I get:

Timing cached reads:   13268 MB in  1.99 seconds = 6665.18 MB/sec
Timing buffered disk reads: 712 MB in  3.00 seconds = 237.17 MB/sec

I would have expected this to run 5 times faster than the old one, not slower. The server is intended to deliver streaming media and so I need super fast access and transfer to keep up with 2 1Gb network ports I hope to max out at times along with performing it's other tasks.

I just put together a bunch of copies of a 400MB MP4 file to get 45GB to copy from one directory to another and it took 96 seconds, which just seems wrong for everything I have ever heard of the performance boost of RAID 0.

It is setup as a hardware raid, is there anything I need to do in Linux to take advantage of the extra speed that should be there? Does it matter which flavor of Linux I use? I am comfortable with CentOS and Ubuntu but could do others if needed.

Is there a different command I should use to measure performance? I tried using iotop and iostat yesterday to monitor the RAID usage and couldn't get it to report any usage while copying 2GB files over FTP, so kind of stuck trying to set a benchmark, comparing it's performance across servers, and monitoring it so I know when the hard drives are maxing out and need to be replaced with SSD.

Solution 1:

Wow... there's a lot to address here.

• Disk performance isn't just about throughput. There's the notion of IOPS and latency and service times to contend with. Most workloads are a bit random in nature, so 25 enterprise disks in an array will always trump 4 disks from an IOPS perspective.

• hdparm is not the right tool to benchmark enterprise storage. Look into purpose-built programs like iozone and fio.

An example iozone command that could be helpful for you is (run from a large directory on the disk array you wish to test): iozone -t1 -i0 -i1 -i2 -r1m -s72g

• The design of this server means that your disk backplane is oversubscribed. There's an expander chip on the server and those 25 disks are sharing a 4-lane 6Gbps connection to the RAID controller. That means that you have a theoretical maximum throughput of 24Gbps (or 3,000 Megabyte/second) to the array. That's a ceiling, and you won't see performance beyond that point.

• Ubuntu is almost never the best choice when hardware drivers and support are concerned.It's not officially supported by the server. CentOS or RHEL would be a better fit for this hardware.

• HP Smart Array controllers have the ability to carve a group of disks (an array) into multiple logical drives of varying capacity and RAID levels. The example below shows a 4-disk array carved into three logical drives. One of the logical drives is configured with a different RAID level than the others.

Smart Array P420i in Slot 0 (Embedded) (sn: 0014380296200A0)

  logicaldrive 1 (72.0 GB, RAID 1+0, OK)
  logicaldrive 2 (1024.0 GB, RAID 1+0, OK)
  logicaldrive 3 (869.1 GB, RAID 5, OK)

  physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SAS, 900.1 GB, OK)

• At no point should you use RAID 0 for a logical drive here. If you can spare the space, RAID 1+0 will perform very well with this hardware combination.

• You have LVM in place. That's not the best approach when working with these HP Smart Array RAID controllers. It's an additional layer of abstraction, and you won't see the best performance (although it can be tuned to work well).

• Firmware. You'll want to update the firmware of the server and related components. Functionality improves with each HP RAID controller firmware revision.

• RAID cache configuration. Make sure the RAID battery is healthy and that the cache read/write balance is right for your workload.

• Filesystem choice. XFS is a good option for streaming media. But the rates you're asking for are relatively low. Any modern Linux filesystem should be fine.

Solution 2:

One thing that jumps out at me right away is the low cached througput on the new server, which indicates you have a ram bottleneck. You mention it has 36 GB of ram in it, which is a bit of an odd number. I am betting that you failed to install the ram in pairs ( or 3x if this cpu/board is capable of triple interleaving, as I suspect it is ), and so the ram is not interleaved properly. You might run memtest86+ to get a good measurement of the memory throughput.

As for the actual IO throughput, for an array this large you are probably not throwing enough requests at it. hdparm only reads 2 MiB at a time, so if the array is using a 512k stripe factor, that will only keep 4 of the drives busy at a time. For other commands like dd, it depends on the readahead size and the default readahead size is only 128 kb. Try using dd instead of hdparm ( without iflag=direct ) and pump up the readahead factor in /sys/block/dm-X/queue/read_ahead_kb to make sure the kernel sends large enough requests to keep all of the drives busy, or if you do use iflag=direct with dd, then make sure to use a very large value for bs=, like 32MiB, and then the kernel readahead setting doesn't matter.