How do you manage servers in a different town

We have servers in two different locations in our town. We connect to them through ssh to do almost everything that we need. But whenever we need to re install a server, we need to drive to the hosting location to do it.

This works well for now, but we are planning on getting a third location in a different city. Driving there won't be an option anymore, so we are looking for ways to remotely manage the servers. We need something that would allow us to reinstall the servers, make hard reboot if the server ever crashes and we can't ssh to it and see the actual booting remotely so we can see what's happening if a server is not coming back up.

So what are you guys using? Can you give me pros and cons about different tools.

By the way, we are currently using Dell servers. But the we are thinking of going with blades for the new location and we don't mind changing to a new vendor.

Solution 1:

You mention you use Dell servers. Dell offers an option in some servers to add a module to the system that you can basically remotely connect too and see the console.

Anyway if the you already have equipment that doesn't support this you should look at getting an IP-based KVM, and manageable power strip. This way you'll be able to remotely access the console and power cycle your equipment.

I suspect I would also be tempted to leave a partition on the disk with a livecd or perhaps you could leave a USB device plugged into each server liveos. The idea being that if you can't do anything remotely you'll be able to connect to the system remotely using whatever OBM method you choose. Once you get connected you'll be able to change the boot order and get to your rescue environment. Another option might be to setup a PXE boot environment so you can netboot your diagnostic tools.

Solution 2:

I managed a web farm on the east coat. I used HP proliants with ILO2 which allowed me to rebuild a server remotely - For eveyday work i had a vpn connection and RDP'd like i would if the servers where local- Instead of ILO you could use KVM's

It pays to have some "hands on" time at your remote data center for doing hardware upgrades or coordinating with vendors - trouble tickets and routine stuff like offsite storage of backup tapes (iron mountain)

There is a bit of latency involved so you need to take some extra care but as long as you have tools like SIM, SCOM etc its pretty easy to centrally manage your servers remotely