What a Linux administrator must know by head?

Are you really sure you care about the day-to-day things? Personally I think the things you should have memorized are the things you will need to do when something is broke, and everyone is breathing down your neck to get the network back up. The day-to-day things tend to vary based on what your Linux boxes are doing on your network.

I think there are a few skills that are pretty important.

• You must be able to configure the network using just cli tools like ifconfig, route, and ip.

  • A few times a client has called saying their a Linux box had failed. I had them boot a livecd. But the server was on a network without DHCP (it was the DHCP). Once the system was booted I need to walk them through starting the network and SSH so I could remotely connect and help them diagnose and fix what was broken.
  • You may be at a point where you cannot access the Internet, and you will need to know how to get online.

• I think you should know how do a full backup of the system using tar, rsync, or dd. If you don't know how to do a backup and restore things you almost certainly should not touching systems. You do need to also actually make sure backups are made before making system changes.

• I think you should know how to access the filesystems from a livecd on your servers. This means you should know how to activate LVM, and Software RAID based drives, access partition information, and mount the file-systems.

  • If your server is unbootable, you may need to access the file-system and fix something. It will be pretty painful trying to figure out how to actually mount things in an emergency. Be prepared ahead of time.
• You should be familiar enough with the boot process to be able to change things at boot. Most systems use GRUB, but you may run into LILO.
  • Importantly, know how boot into different run-levels such as single-user.
• I think you should have at least a working knowledge of how to do some basic captures with tcpdump and be able to read the results. All the nice GUI features in Wireshark are nice, but if something is broken you may not actually be able to access Wireshark.
  • There are a large number of networking problems that I was able to quickly identify and solve just by running tcpdump.

Know What Tools You Have

You'll never know everything ahead of time. But you can know what you have to work with. The more tools you know about, the more you'll be able to use. If you know what the tool is, what it does, and where to find more information about it, then that's good enough to start with.

Get really familiar with the man pages. You don't have to memorize them, but you should know where to find what you're looking for. man pages are better than Google for looking up syntax details, since the pages installed on a given system reflect the various quirks or version-specific information that corresponds to the system your looking at.

If you use apache a lot, then I recommend learning the apache configuration syntax. If you use nginx instead, then learn that instead. But either way, you should know what both are and how they're different.

System Tools

There are a few tools that will help you no matter what type of sysadmin work you're doing. Assuming you know the basics, like chmod, mount, etc., here are a few very helpful tools some admins don't understand well enough:

• rsync
• sar / iostat (part of the sysstat package)
• setfacl / getfacl (most admins think chmod/chown is all you have to work with)
• curl and/or wget
• iptables
• who / last / w

Command line Ninja

I'd say a solid understanding of shell scripting does wonders for making difficult things quick and easy. If you have to look up the syntax, then chances are you won't do it at all, so knowing ahead of time is critical.

For example, let's say you have directory full of mysqldump ".sql" files, each representing a database that needs to be imported into the server. Do you import all 35 of them manually? If you are reasonably familiar with shell scripting, it's really quick an easy to just type one command and then go grab some coffee:

Note: I split it up into separate lines for readablity; if you leave the semicolons in, you can put it all in one line. Otherwise the semicolons aren't needed at the end of each line.

 for FILE in *.sql; do 
   NAME=${FILE%.sql}; 
   mysql -e "create database $NAME"; 
   mysql $NAME < $FILE; 
 done

Also, I recommend brushing up on using sed. Think of it as a way to apply regular expressions anywhere. http://www.grymoire.com/Unix/Sed.html

Say you changed your telephone number and need to update all your web pages accordingly (and save a backup copy in case you mess up).

sed -i.bak 's/555-1234/555-4321/' *.html

Knowing how to properly chain existing tools to do new things can be really helpful as well. Say you need to do the same as above, but also search inside subdirectories --

find public_html -name '*.html' -print0 | xargs -0 sed -i.bak 's/555-1234/555-4321/'

It's also useful to have some experience with perl. You may not need to write any serious programs with it, but it was designed to do a lot of the things that sed and awk do, only perhaps a little more flexibly.

Perl can be used to do command-line magic using the -e option. Using with -p, -n, and -i, you can quickly write simple filters to do really useful things. For example, say you need to find the IP address of everyone who accessed "/admin.php" in September:

perl -ne '
  /([^ ]+).*\[..\/Sep\/2010.*\] "GET \/admin.php / and print "$1\n"' < access_log

See? That wasn't so bad. As the sysadmin, you're expected to know how to do this stuff.

I'm a Windows admin who dabbles a little in Linux, so am not in a position to answer the question directly. However, in my opinion once you hava a decent grasp of the basics the one single most important thing an admin needs to know, regardless of OS, is where and how to find the answers.

In addition to the other answers :

I think you should also know your way around the processes are handled :

• basic knowledge where to find stuff in /proc
• the ps, top, vmstat and some of their more sophisticated spin-offs (ntop, htop, etc.)
• know how to interpret output of at least one good monitoring tool like nagios (might be overkill) or munin.

I think you don't need to master sed (I kow I don't at least), I manage to get by with one of the greps (grep,egrep,zgrep,etc) easily. You have to know basic regular expression syntax, though.

I think you should know basic commands to manipulate and/or monitor MTA (postfix or exim) and MDA (dovecot, cyrus, courier) if you maintain a mail server. Even if you don't run one you'll have to be able to run basic SMTP tests on a MTA, if only for local delivery issues.

You should know your way around the authentication system your using (PAM, LDAP). Where are your passwords stored? using what procedures? What applications use what authentication mechanisms?