Sanity check on 40TB server configuration

Solution 1:

Based on your problem description your issue isn't so much the server as the storage.
You want a reliable, robust filesystem like ZFS that's designed to handle large storage capacity well, and has built-in management capabilities to make that end of the system easier to manage.

As was mentioned in the comments, I'd go with ZFS for the storage pool (probably on FreeBSD because I'm most familiar with that operating system and because it's got a long, proven track record of solid performance with ZFS - My second choice OS would be Illumos, again because of the well-tested ZFS support).

As far as serving up the files I agree - you don't need much in terms of hardware to just push data out the network port. Your primary driver for CPU/RAM is going to be the needs of the filesystem (ZFS).
The general rule of thumb is ZFS needs 1GB of RAM, plus 1GB for every 10TB of disk space it manages (so for 40TB you would need 5GB of RAM for ZFS) -- the relationship isn't quite linear though (there are plenty of good books/tutorials/docs on ZFS that can help you come up with an estimate for your environment).
Note that adding in ZFS bells and whistles like deduplication will require more RAM.

Obviously round RAM requirements up rather than down and don't be stingy: If your math says you need 5GB of RAM don't load the server with 8GB -- step up to 16GB.

You can then either run your server directly on the storage box (which means you're going to need even more RAM on that box to support the server processes), or you can remote-mount the storage to "front-end" servers to actually serve client requests.
(The former is cheaper initially, the latter scales better long-term.)

Beyond this advice the best suggestions I can give you are already well covered in our Capacity Planning series of questions -- basically "Load Test, Load Test, Load Test".

Solution 2:

I use ZFS for a multi-TB server and it has been rock solid. I used OpenIndiana to start with and have now moved to FreeNAS as it does what I need it to do.

I would recommend using an LSI HBA card (9211-8i is a good base card) with SAS expanders (SuperMicro cases can be ordered with integral SAS expanders that are based on LSI chipsets). The LSI firmware is supported in FreeNAS and FreeBSD. Check for appropriate versions (V16 is good on FreeBSD V9.x).

Given the write once read many nature of your system, I would use a ZFS Z2 topology (avoid RAID-5 and Z1 with drives this size). Given that you are using 4TB disks, the rebuild (resilver) time for a large single vDev array would be a long time if the pool is full. To avoid long rebuild times, arrange the vDevs in groups of 6 or 10 to create the pool (recommendations from FreeNAS documentation). A pool made of three 6 drive vDevs (4TB drives assumed) would have a useable capacity of ~ 48TB and offers a good level of fault tolerance (remember you still need to back stuff up as RAID does not replace backups :) ).

To speed things up for commonly accessed files, you can throw in a couple SSDs for L2ARC (likely not needed for your application but they are pretty cheap for 120GB SSDs).

And as stated, use lots of RAM. 64GB is not overly expensive given the other hardware in the system. Unfortunately, the smaller XEON cannot use more than 32GB. You could try it but more RAM would be better according to ZFS literature (I use the XEON you mention with 32GB ram and a 24TB capacity Z2 array and it works fine).

Another advantage of ZFS is that you can setup periodic snapshots. This way, you can restore previous versions easily and the snapshots are very space efficient. Furthermore, you can replicate any snapshot to another dataset (local or remote) and this can be done over SSH for security.

I really like the reliability of the ZFS system. I also like the fact that it is hardware INDEPENDENT!! Any system that can see the drives can import the pool. No firmware dependancies etc. that can happen with hardware raid (not an issue with better cards but they are more expensive than HBA cards and need drivers etc. - been bit by that in the past).

Given this post is older, you likely have a solution. If so, mind telling us what you built?

Cheers,