How to migrate a web site from one server to another with minimal downtime?

I have a server hosting a web site and other services that needs to be reinstalled. I would like to relocate these services to another server temporarily, with as little downtime as possible. Both servers are in the same data center, and can be on the same network switch.

What is the best technique for moving these services with minimal downtime? The site is database-driven, so ideally I want a "railroad switch" event, where I can ensure all traffic is moved to the new server at once. I don't want to have a situation where the old database gets updates after I've migrated the data to the new one.

Two things I have considered:

Change the DNS to point to the temporary service. The major issue here is that I don't control the propagation time for DNS, and other servers can hold on to the cached results for a while, leaving the site "down" for users that get the old address.

Is there a way to fix that problem with Apache + redirects? I suspect not, since name-based virtual hosting breaks without the domain name, which I can't use because it's stale.

Bind the old IP address to the new server and (temporarily) assign the old server a different IP during reinstallation. I can leave DNS alone in this case.

Are there any other simple solutions I am overlooking?

Solution 1:

It sounds like you might best be served with a relatively simple solution ... because you can tolerate a bit of downtime. I would avoid fooling with DNS, because you have little control over the propagation/caching delays.

1- build temp server
2- bring down services on primary server
3- move/copy key data from primary server to temp server
4- change primary server to another IP address
5- change temp server to primary IP address, bring up
6- fix primary server (on different IP)
7- bring down services on temp server
8- move/copy key data from temp server to primary server
9- turn off temp server
10- change primary server back to primary IP address, bring up

The only downtime is when the data is moved between servers, and will vary depending on how the data is moved.

Note: if you have a firewall and are doing NAT, changing the NAT between primary and temp is a good alternative to swapping IP addresses and will reduce the downtime.

Solution 2:

Providing there are no other services bound to the IP then go with switching that over. It doesn't take long and you can be absolutely sure that traffic is going to the correct destination.

Just be aware of neighbouring machine's ARP caches. It's good practice to use arping -s after the change.

Solution 3:

If you have Lan speed connectivity between the 2 systems and full access, using drbd (drbd.org) may be a good option to get the data sync'd between the systems before a cutover and back.

Setup DRBD and let it sync
Shut down db & web server
Switch drbd on original machine to secondary
Switch drbd on second machine to primary
Change original server IP
Add old IP to new server
Bring up db and web server on secondary system

Flip them around when the original system is rebuilt

The option to use database replication is good also if your "data" is primarily in the db

Waiting for DNS propagation even with a low TTL will provide 'inconsistent' results