I have 23 Web Servers - I need to wipe all the servers - without loss of service on any sites hosted across them

Okay so it's a challenge and I'm having trouble planning it mostly due to lack of experience. However it's become a task in my job and I'm doing it mostly solo with an extra hand.

I'm comfortable with Linux and all these servers are Debian or CentOS. Some are VPS, Shared hosts and other are Cloud VMs. I'm new to the company and have only gotten time over the last several monnths to take stock of outstanding issues. They all run plesk or cpanel.

I've found these servers were never configured, just bought and sites thrown up on them with very basic administration in the last two years (disk space clean up and domain/hosting management tasks mostly). There was no iptables set up and no security precautions taken and so these servers are riddled. I figure the best solution here is to start fresh and consolidate our hosting to fewer servers as they had just purchased new shared servers willy nilly. I feel we can support our entire client base on around 1/4 of our cluster.

My problem is that I'm not sure how to plan this move. It will require a LOT of moving sites and DNS changes. I'm thinking it would be wise to seek advice from any wizards out there who deal with this sort of thing regularly or at least have the experience to point me in the right direction.

Mud

Solution 1:

There's no simple solution to this - its probably going to be a long and annoying road to getting everything moved. I'd start with a few things, but the book I recommended in the comments likely goes into a few details. I see four main stages here - inventory, redeployment, testing, and acceptance.

• What am I hosting right now

If you haven't already, do a complete inventory of what you have and what you're using now. Identify what are critical resources, and what are just sitting there. This question on SF may be a good start, though naturally, you will need to adjust it for a debianisque environment - maybe like this. This is the most critical stage.

'Web' servers are not all the same - check if you're running some process that runs its own web-server (like django) or if its a standalone webserver like apache.

• re-evaluate the current setup for best practices

Are you using EOLed versions of your OS and other software? Can you migrate? What issues would you face? Are there any known security risks? Can I standardise everything to a single version of all my software? Can I automate my deployment processes and to what extent (look at puppet and chef).

• create a set of tests for each server

Make a list of what needs to work, and how you're going to test it

• decide what you're moving to

That way you can pick the best P2V or V2V process for your needs.

• decide what to consolidate

You may choose to run several set of services that were seperate before on one server. You may also choose to run VMs. Both running very task specific VMs (you can restart one without affecting other things), and grouping together similar tasks (so you have fewer logical systems to maintain) have advantages

• set up an isolated testing environment

This way you can set up and test a system move without it getting in the way of production. You're likely going to be able to do this on maybe a single server running VMs.

• have a staging area for tested systems

Move the VMs over to the staging area and test them one last time

• Make use of maintainance windows for cut-over to the new systems.

Set a short dns ttl for systems that have been migrated successfully and change the domain name and other relevant settings during a maintenance window. Test again. Keep an eye out for any errors once the new server has been deployed

• Do not destroy old server installs until you can verify the new one works

Cover yourself. If something does go wrong, in the short term, you can always fall back to the old server.