Adding 60TB Storage to an SLES 10 Server

At my previous job we had a similar problem. We were doing production for planetariums and each frame was 64 MegaPixels. Lots of large images. These would be processed for each theater in a very aggressive read operation over a cluster of computers.

The server in this case had a similar storage setup. Multiple external direct attached RAID arrays. Each of these were in RAID6 volumes exposed to the host and added to a VG (Volume Group) under LVM (Logical Volume Manager). Each show/production would then get their own LV (Logical Volume), formatted XFS, which we would grow with the project as required.

If your datasets are pretty static or grow in a predictable way like this then this approach should work well for you. But be careful this approach does have a downside. You end up having to micro-manage the LV's on your storage. Some admins prefer it this way but others would try to avoid it. But this allows you to grow each LV and XFS file system as the dataset grows. Keeping your XFS volumes as small as possible so that you don't get stuck with a fsck that takes years to complete. And can act as damage control should a file system go south.

Disclaimer: If I were to set this up today I would use OpenSolaris and ZFS. Mainly b/c it avoids the micro managing problems and is a superior file system/volume manager. So you may want to have a look at that as well.

I would be so much more included to buy more disks and RAID 10 them.

I've had horrendous problems with hundreds of 1TB FATA (fibre-attached SATA) disks we bought a while back, these are £1k each and I'm losing 5% per month! Basically they're just not designed for a 24x7 duty cycle and that you may have the same issues is why I'd recommend R10.

RAID6 is a step in the right direction but if you have the ability I'd leave at least one disk aside as a hot-spare - if a disk dies anywhere on your array it'll jump in and stripe while it waits for you to replace the failed disk. On that subject make sure you have at least 2 or 3 spare disks on-site ready for replacement and also make sure you have all the alerting setup to let you know when there's a problem 24x7.

As for performance, well those 2GB disks aren't that slouchy for a 7.2k disk and SAS can be very quick so I'd expect the 70MBps for sequential reads you mentioned - obviously randoms and writes will be pretty low.

Sorry if I seem negative, I've just wrestled with storage for years now and can only sleep easily with enterprise disk systems - I've just pulled too many 48/72-hour shifts fixing lower-end gear.