Very poor performance on LUKS/LVM/RAID combination under Debian Squeeze

I have the following issue: I have an encrypted partition on a lvm on a plain partition on the hd. Now I craeted a RAID array to increase the performance. That means, that I have the following stack: HD(s) -- partition(s) -- RAID -- LVM -- Cryptsetup/LUKS. Now the performance is more or less the same with and without RAID (some measurements below).

Can anybody give me a hint why the performance does not gain a boost?

The measurements: First the output of hdparm -t ...:

/dev/server-multimedia/pics:
 Timing buffered disk reads: 208 MB in  3.00 seconds =  69.22 MB/sec

/dev/mapper/pics:
 Timing buffered disk reads: 198 MB in  3.01 seconds =  65.77 MB/sec

/dev/server_raid/pics:
 Timing buffered disk reads: 860 MB in  3.01 seconds = 286.09 MB/sec

/dev/mapper/pics_test:
 Timing buffered disk reads: 204 MB in  3.00 seconds =  67.98 MB/sec

/dev/server-multimedia/pics is the partition without raid and encryption. /dev/mapper/pics the partition opend by luks. /dev/server_raid/pics the raid based encrypted partition and /dev/mapper/pics_test is the luks-opened, raid based partition.

We can see that for the plain partition we do not loose much at all. Where the raid based partition seems to have a very bad performance.

I also checked the cpu state while the test ran. First with the plain partition:

top - 13:07:41 up 5 days,  2:41,  4 users,  load average: 0.22, 0.27, 0.16
Tasks: 287 total,   2 running, 285 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4%us,  1.2%sy,  0.3%ni, 95.1%id,  2.8%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:    900980k total,   752636k used,   148344k free,   178452k buffers
Swap: 26364332k total,   116044k used, 26248288k free,    95880k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25592 root      20   0     0    0    0 R   92  0.0   0:24.13 kcryptd
 6159 root      20   0  3824 3820 1492 D   20  0.4   0:00.83 hdparm
25591 root      20   0     0    0    0 S    4  0.0   0:00.38 kcryptd_io
 6168 christia  20   0  2612 1176  796 R    2  0.1   0:00.02 top

Now with raid:

top - 13:07:54 up 5 days,  2:41,  4 users,  load average: 0.25, 0.27, 0.16
Tasks: 287 total,   3 running, 284 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4%us,  1.2%sy,  0.3%ni, 95.1%id,  2.8%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:    900980k total,   619508k used,   281472k free,    86760k buffers
Swap: 26364332k total,   116044k used, 26248288k free,    55952k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
5594 root      20   0     0    0    0 R   84  0.0   0:06.76 kcryptd
6159 root      20   0  3824 3820 1492 D   18  0.4   0:02.94 hdparm
 404 root      20   0     0    0    0 R    4  0.0 102:20.06 md1_raid5
5593 root      20   0     0    0    0 S    4  0.0   0:00.39 kcryptd_io
6170 christia  20   0  2612 1180  796 R    2  0.1   0:00.01 top

I can see a quite high CPU load. But in the top lines most of the time the CPUs seems to do nothing (approx. 90% idle). So what? Is my CPU the bottle neck?

That looks like you have a CPU with 4 cores (or at least Linux treating it like 4) and dm-crypt takes one core completely without being able to use the others. If the CPU does not allow more than 70 MiB/s then increasing I/O speed does not make a difference, of course.

I am surprised, though. dm-crypt shall be multithreaded since kernel 2.6.38 (that was March 2011). Maybe you can increase the throughput by configuring a different cipher. Or you get a CPU with AES-NI (crypto in hardware, unlimited speed...). What is your cipher (cryptsetup luksDump /dev/... | grep Cipher)?

Edit 1

I just found this on http://code.google.com/p/cryptsetup/wiki/FrequentlyAskedQuestions:

Since version 1.60 cryptsetup supports the "benchmark" command. Simply run as root: cryptsetup benchmark

I have 1.4.2 (kernel 3.4.33) so I could not give that a try.