How do you set up redundant servers?

To the sysadmins out there, I'm trying to get an idea about how you go about maintaining redundant servers for small projects. The modest number of servers in my mind is two, and three main essential services come to mind: HTTP, mail and DNS. How do you automate this duplicity? Is rsync the tool of choice (again, for small projects)?

In addition to common tools for these tasks, references to books and articles would be greatly appreciated. The more hands-on the approach, the better.

It sounds like your real question is actually how to make a service highly available (ignore if that is not the real question). Since you are talking about servers, services and data (zone files, apache docroot, ...) -- if my guess is right you might want to adjust the title and tags of your question :)

Redundancy (as in multiple copies of the data)

can be achieved with:

• drbd (synchronous -- gives you certain guarantees that on both servers there'll always be the same data)
• rsync (asynchronous -- no guarantees about data consistency between 2 hosts)
  • scp or any other tool that can copy data between hosts (or even cp if you NFS mount)

I believe that is not what you're actually asking for...

High Availability

rsync is not the tool of choice for high availability (HA).
It is the tool of choice if you want to synchronize 2 directories.

You should walk thru the Clusters from Scratch Guide from Clusterlabs to get an idea. This covers the following topics:

• HA of Services
• Migration of Services between Hosts (so that any one can go down and you are still providing service)
• Sample Services
  • Apache (without any synchronization for the data)
  • drbd (syncrhonous block device synchronization)
  • GFS (Cluster File System that allows the same block device to be mounted by different hosts)
• Active/Passive and Active/Active Clusters

It's more or less a step-by-step howto that will give you the right pointers. But be aware that HA is not a topic to get right in one afternoon if you just started with it. Also since you tagged it with sysadmin-basics: expect having a hard time (but it will pay off after understanding the concept)

Redundancy isn't usually something you just slap onto existing systems with rsync. For HTTP it's common to use a load balancer that supports failover such as HAProxy or MS's NLB, or possibly an IP sharing solution like CARP; the backend server would serve from a common shared storage, or two file system synchronized with something like rsync, or a database system (again, could be more or less complicated). For e-mail, it has it's own systems in conjunction with DNS to provide round robin and preference order for SMTP; multiple POP or IMAP servers to front a redundant DBMS; or Exchange has DAGs. DNS also has its own mechanisms, mainly multiple name servers in master/slave configurations (or multimaster in some cases).

How you achieve redundancy in a particular situation, with particular software, is highly dependent on your requirements; certainly not a one-size-fits-all that I can point you to a quick how-to document.