What "pieces" are needed in order to set up a cluster of physical servers?

Background:

Currently, we use Rackspace cloud servers. We have no intention to stop using them, but would like to look into setting up a cluster of physical servers (probably desktop computers in the $400 range with 8gb memory each) to offset some of our load and work as a secondary, more powerful, less reliable system. To put things in perspective, we can buy comparable desktop computers for the same price as we pay in one month to rent them on Rackspace Cloud.

I understand that this is generally a dumb idea. However, in this particular instance, the server cluster is needed for its computation power. It is not mission-critical, it does not host a consumer-facing website, and if it goes down for a day or two, its not really a problem.

Currently, we have access to business class verizon fios. If I understand correctly, we can get at least 25 dedicated IP addresses with this service, this should be enough.

Requirements:

• Each server runs Linux Centos 6.3
• Some of the servers run Python and execute processes from a task queue (Redis or RabbitMQ)
• Some of the servers are capable of serving static files and Python driven REST APIs
• Some of the servers host a Cassandra database cluster
• One or more of the servers are a Redis database servers
• One or more of the servers are PostgreSQL servers

Questions:

What kind of router or switch is needed?

• We would like the computers to be able to communicate effectively with each other via internal IP addresses. This is especially important for communicating with servers hosting Redis that need to be able to respond to requests very quickly. Are there special switches or routers that need to be used to connect the servers together?

Are Desktop computers ok for this?

• We have found that we are mostly RAM-bottle necked, I understand that some servers have highly superior CPUs, but I'm not sure we need CPU power as much as we need RAM, which is cheap in Desktop computers.
• Will we have problems with the WIFI cards in the desktops or any other unexpected hardware limitation?

What tools should be used to "image" the servers.

• For example, when we get an installation right for a Redis server or Cassandra node, are there tools that come with Linux Centos 6.3 to image the server to a USB drive or something like that?
• Or do we need to use some other software for this?

What other things are we missing that we should be concerned about?

Solution 1:

• You do not need internet IP for that - you can set up a site to site VPN and keep all private. This is what I do with the current 2 locations (my basement in Poland, which has a 2 rack server setup now, and our outside site in chicago). I use 2 routers (virtual in chicago, physical here) to establish private routing.

• Desktop is ok, but depending what you do you may want to go with E3 Xeons - 32gb memroy is possible there, but more important: ECC memory. I really dont want my simulations having bad result due to memory flipping. As explanation: we assign 1gb / core, this is a queue controlled HPC cluster, doing in our case CPU intensive simulations.

• If you are RAM bound - look for AMD. Their G32 socket has an 8 core setup that can handle for a relatively low price 64gb ECC memory. Good for virtualization also. I am getting some of thsoe for an upgrade for my main servers (hyper-v platforms, serving also as domain controller etc.) due to the superior RAM.

• Switch - all nice and dandy. Depending what you do you may want to get one of hte netgear switches that has 48x1gb and 4x10gb - and use the 10gb to link up to your file servers. We just do that - because with the now coming 2x computers (we just add 16 in the blade center) the server side would overload network wise during certain operations.

Otherwise you really need to understand that technology has evolved - switches etc. are reliable today and performant. Nothing special is needed. Network setup will be non-trivial, but not really very complex either.