Samsung 850 EVO + HP P410 limited iops?

I´m experiencing a weird write iops limitation with a HP P410 256mb cache controller and 4x consumer grade ssd´s (Samsung EVO 850) in RAID5/10. After upgrade to latest firmware (6.64) for HP P410 controller, it continues with problems.

Iostat output:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    1.00  158.00    36.00 15631.00   197.07    12.16   75.84    6.00   76.28   6.30 100.10

158 writes/s and 100 % util.

My hpacucli output:

=> ctrl all show config detail

Smart Array P410 in Slot 2
   Bus Interface: PCI
   Slot: 2
   Serial Number: PACCR9SYLZ34
   Cache Serial Number: PACCQ9SYP5CK
   RAID 6 (ADG) Status: Disabled
   Controller Status: OK
   Hardware Revision: C
   Firmware Version: 6.62
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 15 secs
   Surface Scan Mode: Idle
   Queue Depth: Automatic
   Monitor and Performance Delay: 60  min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 0 secs
   Cache Board Present: True
   Cache Status: OK
   Cache Ratio: 100% Read / 0% Write
   Drive Write Cache: Disabled
   Total Cache Size: 256 MB
   Total Cache Memory Available: 144 MB
   No-Battery Write Cache: Disabled
   Battery/Capacitor Count: 0
   SATA NCQ Supported: True

   Array: A
      Interface Type: Solid State SATA
      Unused Space: 0  MB
      Status: OK
      Array Type: Data



      Logical Drive: 1
         Size: 698.6 GB
         Fault Tolerance: RAID 5
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 768 KB
         Status: OK
         Caching:  Enabled
         Parity Initialization Status: Initialization Failed
         Unique Identifier: 600508B1001C8929D0C6820FB103773E
         Disk Name: /dev/sda
         Mount Points: None
         Logical Drive Label: A516FD0DPACCR9SYLZ3435C0
         Drive Type: Data

      physicaldrive 1I:1:1
         Port: 1I
         Box: 1
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: Solid State SATA
         Size: 250 GB
         Firmware Revision: EMT01B6Q
         Serial Number: S21PNSAG270000F     
         Model: ATA     Samsung SSD 850 
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 30
         Maximum Temperature (C): 70
         SSD Smart Trip Wearout: Not Supported
         PHY Count: 1
         PHY Transfer Rate: 3.0Gbps

      physicaldrive 1I:1:2
         Port: 1I
         Box: 1
         Bay: 2
         Status: OK
         Drive Type: Data Drive
         Interface Type: Solid State SATA
         Size: 250 GB
         Firmware Revision: EMT01B6Q
         Serial Number: S21PNSAG269993V     
         Model: ATA     Samsung SSD 850 
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 32
         Maximum Temperature (C): 70
         SSD Smart Trip Wearout: Not Supported
         PHY Count: 1
         PHY Transfer Rate: 3.0Gbps

      physicaldrive 1I:1:3
         Port: 1I
         Box: 1
         Bay: 3
         Status: OK
         Drive Type: Data Drive
         Interface Type: Solid State SATA
         Size: 250 GB
         Firmware Revision: EMT01B6Q
         Serial Number: S21PNSAG269984Z     
         Model: ATA     Samsung SSD 850 
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 31
         Maximum Temperature (C): 70
         SSD Smart Trip Wearout: Not Supported
         PHY Count: 1
         PHY Transfer Rate: 3.0Gbps

      physicaldrive 1I:1:4
         Port: 1I
         Box: 1
         Bay: 4
         Status: OK
         Drive Type: Data Drive
         Interface Type: Solid State SATA
         Size: 250 GB
         Firmware Revision: EMT01B6Q
         Serial Number: S21PNSAG269998N     
         Model: ATA     Samsung SSD 850 
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 31
         Maximum Temperature (C): 70
         SSD Smart Trip Wearout: Not Supported
         PHY Count: 1
         PHY Transfer Rate: 3.0Gbps


   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 250
      Device Number: 250
      Firmware Version: RevC
      WWID: 500143800694906F
      Vendor ID: PMCSIERA
      Model:  SRC 8x6G      

Someone has experienced this problem with HP P410?


Solution 1:

There are a few issues here.

  • You don't have any controller write cache or a battery-backed or flash-capacitor.
  • The HP Smart Array P410 controller is limited in IOPS capacity. It's not a good match for SSDs.
  • Using SATA drives on a Smart Array P410 causes the interface speed to downshift to 3.0Gbps instead of 6.0Gbps.
  • Please try with RAID 1+0. RAID5 is not good for this combination of equipment.
  • Also, your RAID5 parity initialization failed, according to your hpssacli output.
     Parity Initialization Status: Initialization Failed

HP's best practices for SSDs on that controller are listed here.

Also see: SSD (Intel 530) read/write speed very slow with RAID 10 and HP ACU shows parity initialization failed on Smart Array controller

Solution 2:

MLC SSD really need a fast private DRAM cache for delivering high IOPS values.

While your controller has its own cache and it is enabled, your disk's private DRAM cache is disabled. This is a safety measure, as enabling that unprotected (from power losses) cache can put your data at risk.

Sometime it is safe to reenable it (eg: your disks has power loss protection, or your controller correctly propagate ATA FLUSH / FUA commands to the downstream disks), but you can't be 100% sure without testing. Some RAID controller even do not permit to reenable the disk's private DRAM cache.

For testing purpose, you can try to reenable it and re-test your array, but be very careful to put it into production. When possible, use SSD with power loss protection (Crucial M550/M600 have a partial protection, while Intel S3500/3700/DC series have full power protection).