What should I watch out for in building low-budget storage servers?

Solution 1:

I'd be really wary of building your own computers. It sounds like you're pretty dead-set on it, though. Be sure to observe proper anti-static procedures when handing the equipment. Try to avoid using hard disk drives from the same manufacturing lot exclusively when assembling your RAID volumes. Be sure that the disks you're purchasing were shipped in the proper shock-resistant packaging as dictated by the manufacturer (and not a typically NewEgg "in their anti-static bags thrown in a box with some bubble wrap and peanuts" arrangement).

Since you're going to build your own box you need to stock sufficient spares for hard-to-replace components for the duration of the intended production life of the system. Having spare parts availability is part of what makes the "commercial" options. If you lose a backplane, RAID controller, or motherboard you need to be ready to replace them. You're acting as your own hardware support vendor. That's going to increase your purchase cost, but you should acquire spares now while they're still readily available.

I'd shake-down the hardware pretty severely before I even thought about using it in production. Any load-testing or production use simulation you can throw at the hardware is a good idea. You definitely want to shake out any "infant mortality" in the hardware before you go into production.

A "mirror" without multiple prior generations is a poor off-site copy. If you're doing the "mirror" to allow you to put the backup file server into production in the event of failure of the production then it makes sense to keep such a "mirror" up. It's not a backup, though, to my mind. That sounds more like a redundancy mechanism than a backup mechanism. I'd strongly recommend using a backup that allows for storage of multiple generations of changes on the remote server, even if you do also maintain a functional "mirror" of the data on the remote server for production failover purposes.

I'm glad to hear that you're still going to use tape. I'm a big fan of keeping backups off-site and offline. Offline backups are nice because their integrity (provided they're taken properly to begin with) is much easier to insure than a system that remains online all the time. It's pretty hard to attack the data on a tape in a locked box remotely.

Tape is also a great storage medium for archival storage. The cost to expand your retention windows is incremental (as opposed to adding more spinning media to a purely hard disk-based solution). I think you should strongly consider using tape to retain multiple generations of all of your data off-site and offline. Be sure that you're fulfilling any business requirements for maintaining long-term archives on tape and plan to purchase additional media as necessary. You can keep those archives in a more geographically diverse location to allow for some recovery capability in the event of a major physical disaster.

If you're doing a backup that's truly a "copy" then you're not going to lose archive bits. That depends on the software you're using.

Solution 2:

Here's a suggestion regarding your backups that may help to reduce your storage needs:

Perform a Full SQL and System State backup every day (unless you're comfortable performing point in time SQL backups).

Switch to performing incremental backups on a daily basis for your file server backups. I've been in the IT field for 11 years and have yet to encounter a scenario (knock on wood) where I've needed a differential backup to restore a system or to restore data. I may need to go back through a few days of incrementals to find the "version" of a file that a user needs restored but that's rarely the case as the user usually wants the latest backup copy restored. In the event of a complete system failure, the worst case scenario is that you'll need to restore multiple incremental backup sets to bring the file server back in "sync". This will of course extend your recovery window, and that may or may not be acceptable to you and the organization.

I approach systems management from a "possibility vs. probability" point of view and in my experience, while it's possible that I may need a differential backup, it's not probable... from my experience. This has allowed me to reduce the size of our daily backups and the backup window as well.

Solution 3:

There is appliance stuff with replication or at least remote differential copying features based on things like open-e, Nexenta or Windows Storage Server out there in the market.

On a whole, I'd rather not buy components to build them together into servers, but buy servers/appliances with system warranty - they would not cost significantly more, but might save you a lot of trouble in the case of a hardware defect.

Further, you should not use the tape "for SQL Server and system state backups" but to take copies of your critical data off site. You should see to it that you keep a history nonetheless - probably incorporating the SQL server into your standard backup process is a good idea.

If you are a Microsoft shop and eligible for academic licensing, you might want to look into Data Protection Manager to replace your aged BackupExec version - it would get you interesting features and good integration with Microsoft software.

To reduce backup time/stress on the primary, I'd like to take the nightly backup from the mirror. But will copying to the mirror destroy the archive bit I need to get good differentials?

Yes, it will. But differentials are a bad idea anyway - you probably don't need them. Take full backups at reasonable intervals and store them off site. Your backup strategy is primarily there to alleviate the effects of a catastrophic failure - if both of your campus IT sites are destroyed at the same time by a catastrophic event, it probably does not matter much if your users have lost one day or five days worth of data - as long as not the entire data is lost.

I'm looking at 44TB of raw disks space, but only exposing 4TB to my users after carving out RAID and backup space. Can I safely do better than that 11:1 ratio?

Depends on your security and performance requirements. You probably can save the cost for a RAID 10 setup on your "backup" mirror - as long as this is not meant to serve users directly, you can use a less performant and less expensive striping-with-parity setup (RAID 5, 6 or Z).