Cost effective and frugal SAN solution for under $5000?

Openfiler can provide the iSCSI infrastructure for the SAN. Your cost then is simply the disks and enclosure.

You might look at something like this:

http://www.newegg.com/Product/Product.aspx?Item=N82E16811219021

which provides 20 hot-swap drive bays for SAS drives. Slap in a motherboard of your choosing and some good SAS controllers. Let openfiler handle the RAID-6 and the iSCSI for you. I think you could do it quite cost effectively.

Don't have any hardware recommendations off the top of my head, but as for software and networking you should definitely go for iSCSI. It will allow you to run the SAN network over your existing ethernet infrastructure, and adding more clients is basically free if you use the open-iscsi initiator.

Of course, having a dedicated switch or at least a VLAN is recommended, as well as a dedicated port on each machine for iSCSI traffic. You can get by without this but performance will suffer.

For the target, you can run OpenFiler, which will basically give you an appliance-like solution and ease the burden of configuration.