Best RAID setup for multimedia fileserver? [closed]

Solution 1:

I agree with what @sysadmin1138 said in his answer and voted it up but I have a few comments and my experience I'd like to add.

Prior to doing IT fulltime I ran a small graphics department for 8 years dealing with this sort of stuff. We had 3 artist and close to 50 node cluster of rendering computers creating and processing image sequences, video frames, up to 64 Megapixel per frame. We were a small company working in a niche market so budget was always an issue for us as well.

Keep in mind that a video server for film and multimedia with 3 users can mean many things. It can mean 3 users streaming the same web quality video or it could mean 3 users compositing shots with a dozen or more layers of 4k uncompressed or lossless compressed frame sequences. The type of network and storage requirements are vastly different in these two examples. So it's hard to really give you solid advice but I'll mention what setup we used at my previous job as it's beyond the high end of that spectrum.

Our file server/storage setup was comprised of many external RAID units from Aberdeen attached to one of their servers running Linux. We chose Linux b/c we were comfortable with it and could avoid the costly Window CALs. Hey they add up when you have a sizable rendering farm. But Aberdeen also sells their servers pre-configured with Windows. They also recently started offering some ZFS NAS/SAN storage servers that are running Nexenta which may be worth a look as well. But their hardware comes with a 5 year warranty so I would highly recommend doing something like that over building something yourself. Monitoring wise their DAS units have an Ethernet management port with a web interface for configuring. But they can alert you to any issues via email and snmp.

The Aberdeen DAS units were exposed to the server as multiple RAID5 LUNs at first but we eventually migrated all of them over to RAID6 LUNs and we were using LVM to manage the storage as a pool. Each project got it's own logical volume which would grow as the project grew. This made it easy to archive the projects as they were completed and then return the space back to the pool. The closest thing to this on Windows without going to 3rd party filesystems and management tools like Veritas is sysadmin1138's suggestion of using Windows Dynamic Disks. In our case RAID10 may have been a better option but our budget wouldn't allow us to give up half our capacity so we chose to stick to RAID6. The server was able to saturate the 2 Gbit links it had at the time so I don't think that choice slowed us down that much.

But I started that setup in 2003 with 3TB of storage and they have been growing with it since. Now it's nearly 8 years old and most of the components have been replaced over the years. But it's pretty simple to add additional storage to it and migrate logical volumes off the older units to decommission them. A couple years back they also replaced the file server part of the setup. Their current server has 4Gbit links. Storage wise the last I heard they were a little over 40TB.

A few other thoughts.
I know it's tempting to go with 2TB drives as it gets you where you need to be capacity wise pretty easily but it comes at the expense of fewer spindles which means fewer IOPs and longer rebuild times when a drive dies.

Solution 2:

I'm not going to give you pricing since that's so highly localized as to be meaningless.

Looking purely at the storage usage patterns you're proposing, the biggest I/O driver I'm seeing is the video and graphic design. Video tends to be very large. Large enough that you might run into network I/O bottlenecks before you run into disk I/O bottlenecks. It should be snappy for your users, and with enough overhead that when someone is committing a 1.5GB video file to the filer people can still save other files.

As for OS and server hardware, considering your background Windows will work just fine here. The server needs to be 64-bit and with a 64-bit RAID card (if you buy new, this shouldn't be a problem). 8GB RAM is the minimum I'd go with.

Now for the storage infrastructure.

Key drivers:

• You want to add drives later.
  • Expansion of initial LUNs would be highly desirable.
  • Failing that, the ability to add new LUNs to a Dynamic Disk set.
• You want one drive letter.
• You need mirroring.
• You need drive-health monitoring.

Go with a high end RAID card that supports RAID10 and live expansion. Or if costs start to eat you alive when it comes to buying disks, RAID50 and a lot of battery-backed or flash cache. Pay attention to 'Advanced Format' support (4KB sectors vs the previous standard 512B sectors) on your RAID card; if you buy from a big name (Dell/HP/IBM) this shouldn't be an issue, but if you're assembling from parts this will be a big issue.

Once you have your storage infrastructure in place, you'll be relying on the RAID card to present a single huge LUN to the operating system. Since you're 64-bit end to end, you should have no problem with a single 30TB LUN. Adding space should be as easy as importing drives into the RAID config and then Extending the LUN to fill the new space, all of which can be done online. Just keep in mind that a single volume that large will take an age to chkdsk if you ever need to do that.

If you buy from a big name they all have tools/agents that'll monitor disk health, you just have to remember to install and configure them. They usually have a way to send alerts, too. If you're doing it from parts, your RAID card vendor probably has a utility that does much the same kind of thing though may be less featured than the integrated stuff the big vendors distribute.

If you do need to go with multiple LUNs for some reason, keep in mind that Windows allows you to use mount-points. The directory F:\Bigshare\VideoArchive can be its own volume. People map Z: to \\storeserv\bigshare\ and Z:\VideoArchive is that second LUN.

Avoid DFS. Support in non-Windows land is very spotty and can't be relied upon.

RAID50 Considerations

If you have to go there, make sure your drives are enterprise-class. This will increase cost. A 30TB LUN can take a week to rebuild after a drive failure, and you really, really, really do not want a drive failure when you're doing that.

In fact, you would be much better served by making sure no RAID50 set grows past 10TB in size and just go with multiple LUNs; it reduces how long you're out as well as your exposure to a complete data loss event during a rebuild.

Going with multiple LUNs is pretty easy with Windows Dynamic Disks. The type of drive is called a Spanned Drive, and can spread across multiple LUNs of varying sizes. This is how you create a 9.5TB volume on a 32-bit server that can only use up to 2TB LUNs (3x 2TB LUNs and a single 1.5TB LUN). I/O will not be as well balanced as it would with a single 30TB RAID10 LUN, but that's the price you're paying for cheap.

Solution 3:

Up to 30TB with two mirrors per drive? 30TBx3 = 90TB/2TB per drive = 45 drives.

When you start talking about this amount of data (and this number of drives) you also have to think about how you're going to push this amount of data around. Even the initial 10 drives will overrun a single GNIC or RAID card. How about the southbridge on the motherboard? You can chain together enough DAS drives to hit your capacity targets, but it sure will take a long time to do anything with all that data.

If you really need that much mirroring/expansion/reliability I'd recommend an iSCSI SAN. Something like the Dell Equalogic would be a good bet. You'll get a lot further with a bunch of smaller servers/arrays working in tandem, than trying to create one gigantic array.

Additionally: If you can afford it buy enterprise class drives (15k, SAS, WD or Seagate). Since you're buying the drives over time hopefully the cost will be manageable. Running 24x7 consumer drives will be lucky to last two years. So that would be two drives failing every month at your full 45 drive capacity. You'll end up buying a lot more drives because you aren't going to find 2TB SAS drives, but that's not entirely a bad thing. More drives = better throughput.