10GE network: Is it still deadly expensive? Any options?

Solution 1:

I don't think you're not seeing the entire picture here.

You are wanting to connect a file server at 10Gbps speed, which may sound like a sexy idea. The thing you are not seeing is the ability of that server to generate that amount of traffic reading from disks. Getting 1GBps from a file server is, today, a very good achievement. 10Gbps will not only be expensive as you have realized yourself but at minimum 90% useless.

Your best option is to start putting in some blazing fast disks in your file servers if it needs to provide such great amounts of IOs. I strongly believe the "affordable" (notice the quotes) path to this is SSD drives in fast RAID configurations (that is RAID10).

As for networking, a 4x1Gbps agregate will do the trick fine and you can even add more later. Watch out for the fact that internal buses (read PCI*) are not always capable of handling multi-gigabit speeds. This is especially true if you are not using server-grade motherboards.

I believe this is your only "affordable" option here. Infiniband cards are not horribly expensive. I believe you can find some for ~150$ but the switch will be very expensive.