What should every sysadmin know before administrating a public server?
Similar to this question on Stack Overflow, what should a sysadmin who is used to private, intranet-type situations know before being the administrator of a public site?
These could be security related things like "don't leave leave telnet open," or practical things like how to do load-balancing for a high-traffic site.
Solution 1:
Every app, every binary, every package that exists on the server is a liability. Subscribe to the 'least bit' principle; if it's not installed, it can't be compromised.
Implement intrusion detection, such as Tripwire or similar, and scan frequently.
Invest in a hardware firewall and only open the ports you need for your application. Do not allow your administration ports (ssh, rdp etc) to be publicly visible; restrict them to approved management IP addresses.
Have backups of your firewall/switch/router configurations at the time of going into production. If one of those devices is compromised, it is significantly faster to recover from by wiping the device's brain and reloading the config than to perform a line by line audit of the config when the clock's ticking.
nmap your environment from the outside frequently to ensure no new ports have opened.
Never trust the internet; make sure whatever it is that you're serving up to the net is a safe as it can be (perform server-side input validation and sanitisation to stop SQL-injection attacks, for example).
Keep on top of your patching.
If you are compromised, rebuild from scratch with freshly downloaded media. You can no longer trust your backups are safe and haven't also become compromised (although tripwire can help with this) for anything other than inert, non-executable data.
Solution 2:
One tool I've found handy for network hardening is nessus
Basically, you set it up on an external server, and it tries to attack your network with a whole whack of known exploits. You can set it for safe mode (where none of the attacks should crash your server), or if you're pretty confident you have everything patched, or can afford to reboot your servers if necessary, for unsafe mode.
Then it will provide a very complete graded report for each machine it can see of what vulnerabilities/weaknesses it finds, and rate them as to severity - and even recommend actions to be taken to address the issues.
Solution 3:
They should know how their backup and disaster recovery system is working and how they will recover the system when/if it becomes compromised.
Solution 4:
This is a bit contrarian, but security-wise I don't differentiate between an internal server and an external server. Sooner or later someone will make a mistake in a firewall, management will insist a server be exposed because of an important client, Betty in accounting will somehow get a vpn client on her infected home machine, etc.
That said, layers are your friend, and you should blacklist by default.
Layers - you should have multiple layers of security. For instance, a hardware firewall and a software firewall. These theoretically serve the same purpose, but having multiple layers protects against mistakes and mitigates consequences of a single layer being exploited.
Another aspect of layering is "homeycombing", which is essentially multiple DMZs. At some point you have to have some level of trust between your machines and of the people accessing your accounts. If you can narrow those points of interaction, you can tightly control the kind of traffic you trust at any point. For instance, if you separate you interface/app servers from your database servers, you narrow the level of trust. If your app servers become compromised, those attackers gain a minimal foothold to your infrastructure (that is, in order to continue their attack and attempt to exploit your other servers, they only have those established trust points to use).
Regarding blacklisting by default, you should basically shut everything down and demand (even if it's only of yourself) justification for every port you open, username you allow access, app you install, etc.
Solution 5:
On systems with ANY public interfaces, ensure your users have secure passwords by implementing a secure password policy, and by testing the password file with a password cracking utility like john the ripper
You can further guard against brute-force password-guessing attacks by blocking IP addresses after several failed attempts. A good tool for this (on linux) is fail2ban