How should we serve files in a small bioinformatics cluster?
Solution 1:
I'm in the field of computer science and I do research in bioinformatics. Currently 746 on Biostars :)
I have been operating the bioinformatics compute facilities for 3 years at a university (about 40 Linux servers, 300 CPUs, 100TB disk space + backups, about 1T RAM total - servers ranging 16 to 256GB of RAM). Our cluster has 32 8-core compute nodes, 2 head nodes, and we are expanding it with 2 more 48-core compute node. We serve the files to the compute nodes over NFS.
I would recommend switching to NFS for your situation.
We considered switching to Gluster, Lustre, and Samba but decided not to use those.
NFS
I have a few main tips about NFS:
- Have a dedicated NFS server. Give it 4 cores and 16GB RAM. A dedicated server is more secure and easier to maintain. It's a much more stable setup. For example, sometimes you need to reboot the NFS server - a dedicated server will not fail your disk accessing computations - they will simply freeze and proceed once NFS server is back.
- Serve to your compute and head nodes only. No workstations. No public network.
- Use NFS version 3. From my experience NFSv4 was more fragile - more crashes - harder to debug. We switched the cluster from NFSv3 to NFSv4 and back several times before settling. It's a local network so you don't need the security (integrity and/or privacy) of NFSv4.
Storage Hardware
Our current cluster was bought 3 years ago so it's not using SAS, but rather has an expansive FiberChannel drives and san controllers. This is changing, all the new storage that we are buying is SAS.
I would suggest considering a SAS storage. SAS is replacing FiberChannel as a cheaper, faster and a better solution. Recently I did research on the different solutions offered. Conveniently the options that we looked at are documented of Server Fault: What are SAS external storage options (Promise, Infortrend, SuperMircro, ...)?
We recently ordered a 24TB 6Gb SAS - 6Gb SAS storage system from RAID Incorporated. Just for the storage we payed $12k. The order should come in a couple of weeks. This is a no-single-point-of-failure system - all components are redundant and automatically fail over if any components fail. It's attached to 2 servers each using a different partition of the array. It is a turn-key solution so once it's shipped we just need to connect it, power it on, and it will work (RAID6 partitions will be mounted on Linux). The order also included servers and RAID Incorporated are setting-up Linux Debian on those for no extra cost.
Other considerations
Unfortunately, if you do bioinformatics infrastructure operations you probably need to become a storage guru.
For your 10TB partition, pick RAID6 - 2 drives can fail without losing you data. Rebuilding a 2TB drive onto a hot spare takes 24 hours, another drives can fail during that time. I had 2 drives fail simultaneously in a 16 drive array.
Consider dedicating one drive to be a hot spare in the array. When you have more then 16 drives then I would say a hot spare is a must.
Think of a plan of action if hardware fails on the dedicated NFS server. I would keep a twin as a compute node as a potential replacement for the original NFS server.
Finally, I have to mention our file server is running OpenSolaris (sounds unusual - I know). OpenSolaris (as it turned out for us) has excellent server hardware support (FiberChannel, IniniBand, ...). Setting up an NFS server ground up takes 1 hour - all steps a completely straight forward: install os, update through a NAT, setup network, create a zfs pool, create zfs filesystems, share NFS. Sun were the ones who developed NFS in 1984, not surprisingly OpenSolaris is very good at serving NFS. The main reason to use OpenSolaris was ZFS - a good filesystem for bioinformatics. Some features that I like:
- Integrity (all writes are checksumed)
- Pooled storage, snapshots
- NFS exports are configure in the served filesystem
- Online compression
- Reservations (space guarantees)
- Block level Deduplication
- Efficient backups (see
zfs send).
Using Linux for your NFS server would be fine - in that case stick to XFS or Ext4.
Solution 2:
Your budget isn't going to get you very far with SAN class hardware but you should be able to get much better performance by beefing up the hardware you have. Get a decent RAID controller, buy more disks, get a much better switch and maybe a good multi port NIC (get decent server grade ones, like the Intel PRO 1000 GT or ET's).
If your description of the IO pattern is correct you have a 15:85 Read/Write ratio so you will need to go for RAID 10 in order to improve on your throughput numbers with SATA disks. Given your write bias if you were to simply reconfigure your current drives for RAID-5 (or RAID6 which would be more advisable at this scale) performance would plummet. RAID-10 will halve the usable capacity of the disks though.
Getting all of the above, and enough disks to deliver 10TB in RAID10 for $5k is doable, but it's not a risk free exercise. There are some very interesting options described in this question and its answers that are worth considering if you are happy with the risks and comfortable building your own solution.
However my main advice to you would be to start asking yourself (or whoever signs the checks) is how much a storage failure will actually cost your business and whether you are comfortable with that risk. Your budget of $5k may just about allow you to improve performance but you're talking about having 10TB of what I assume is business critical data and processing capacity all riding on an infrastructure with many single points of failure. Now might be a good time to take a long hard look at just how important this infrastructure is and figuring out if you can get enough of a budget together to buy a proper entry level SAN or NAS solution.