Picking up the pieces after missing Linux admin [closed]
Solution 1:
Wow. Where to start.
This is what I would do but hopefully others will jump in with more/better suggestions.
First don't panic. I am assuming you are now root. You are currently the most dangerous threat to the server right now as you have a lot of power and not much idea of what to do with it.
Write down what services the server should be running. You know apache, mysql and postfix are required. I guess you may have an ftp server on there and you are able to ssh in so you need sshd running. Write down what services are installed. The quickest way to find out is probably by listing /etc/init.d/*. You then need to find out what are running. I don't know what the equivalent to the Red Hat chkconfig is, but failing an alternative ps -ef will list what processes are currently running. Also find out if a firewall is installed (such as iptables) and how it is configured.
Next get a list of all the cronjobs that are run. You probably don't need to worry too much about what they do right away but you should have a general idea of what the server is doing at different times.
Again I would write all this down.
Now write down who should have access to the server and who is allowed to have root access. Get a list of users who do have accounts from /etc/passwd.
Do a similar thing for FTP access, and other services if relevant like Subversion or remote MySQL connections.
Now you know a bit more about what your server is doing and who can access it, you should move on to how well it is doing. Check the log files in /var/log, especially /var/log/messages and spend some time looking for any errors.
Check if there are any outstanding updates to do using apt-get update && apt-get upgrade
When prompted to update choose no for now.
So far you should have made no changes.
You now need to review the information you have collected and decide what (if anything) needs fixing. The priorities are attempted cracking attempts in /var/log/auth.log, shutting down uneccessary services and tightening the firewall.
Make copies of all files before you edit them and test changes often so you can easily back out if something breaks.
Backups
You will need to decide what needs to be backed up. Obvious candidates are databases, /home/ /etc/ /var/log/ /var/spool/cron/ /var/www/ and any custom scripts called by crontabs. Then most people write a shell script to back this up locally and then use something like rsync to copy the files to another machine USB drive.
Day to day duties will include, checking the log files for any problems (check out logwatch to help you), performing security updates, checking backups and goinf forward setting up monitoring like MRTG and Nagios to eventually take the heavy lifting out of being an admin.
I would not worry too much though. It may seem daunting but that is because you are asking for it all in one go. The server is probably fine as it is, keep an eye on the logs and apply updates as they are released, plan what you want to do and work towards it, take little steps and try to enjoy it.
Solution 2:
Some things to get you started:
get somebody who has already done such an audit - or at least has some knowledge about the common pitfalls maintaining a server. Seriously - it pays off.
Back up as good as you can and try to reconstruct a spare server - might be a virtual instance somewhere - until you're confident that a) you have backed up all important stuff and b) you're able to reconstruct the spare server from your backup in as little time as you desire. To add karma: exchange the current production server with your spare server. As long as you've not demonstrated that you can reconstruct from your backup, act as if you have none.
Update, read security notices, keep an eye on the logfiles and automate the heck out of this once you know what to look for.