What are the meanings of /usr/sbin, /usr/local/sbin and /usr/local/bin?

1. Directory structure

This should be covered in the Filesystem Hierarchy Standard (2.3 PDF)

/bin/       Essential command binaries that need to be available in single user mode;
            for all users, e.g., cat, ls, cp

/sbin/      Essential system binaries, e.g., init, ip, mount.

/usr/bin/   Non-essential command binaries (not needed in single user mode); 
            for all users

/usr/sbin/  Non-essential system binaries, e.g. daemons for various network-services.

/usr/local/ Tertiary hierarchy for local data, specific to this host. 
            Typically has further subdirectories, e.g., bin/, lib/, share/

2. Installation

I use a package manager wherever possible (e.g. yum or apt-get). This is possible for a very large number of applications, in a few cases you may have to add a repository. My second choice would be lower level packages such as RPMs and compiling from source would be my last resort (but some people prefer this)

Some package managers can install from RPMs (e.g. yum install oddity.rpm)

If you are compiling from source, its probably not a huge step to create your own package so that the system installer knows what you've done.

Then your problem reduces to e.g. yum remove packagename

The alternative is to keep good documentation about all sysadmin activities undertaken (I keep a journal in a text file anyway)

Stuff in all the */sbin directories tend to be only useful for system administrators. You can keep them out of your PATH if you're a normal user.

The different directories don't make much sense if you have a single unix machine on a single disk, but make more sense if you have a big system and different partitions. Remember that a lot of these habits were made in the 80's and 90's when systems were a bit different.

/sbin tends to be very small. These are utilities that you need when you're really hosed. You'd put this on a minimal root partition with /root and /lib. Things in /sbin used to be all statically linked, since if your /usr partition is hosed, any dynamically linked apps are useless. fsck is here and statically linked. If you have a dependency on /usr, obviously you can't fsck /usr/. Of course, if root partition is hosed, you're very screwed. This is why this is such a small partition - lower the odds of a bad disk block by using very few blocks here.

/usr/sbin binaries are general sysadmin tools where you can at least get to single user mode and mount all your volumes. They are allowed to be dynamically linked.

The separate partitions for /sbin (well, /sbin on / partition) and /usr also make more sense when you remember that backup was very expensive in both time and for tape. If they were on separate partitions, you could schedule them differently.

/usr/local can be a network filesystem. So locally written sysadmin tools that can be shared across many machines sometimes go into /usr/local/sbin. Obviously no network fixing utils can go there.

Again, a lot of things made more sense on big machines in a networked environment on managed machines with multiple volumes, less so with one Linux machine on a single root partition.

You really should have your second question be a separate question here on Superuser. It's unrelated to the first.

Yes, having files all over the place sucks. That's why there are many packaging solutions. RedHat created RPM which is used everywhere. Solaris had their package format. HP/UX had theirs, and there's apt and many other package formats. Keep things in the right places (/usr/bin, /usr/lib) as appropriate, but allow easy addition and removal.

For source, there used to be tools that would let you configure and install in a subdir of /usr/local and it would handle symlinks to /usr/local/bin for you. Since the wide proliferation of package tools, this is less necessary, and I forgot their names.

Some people like to install in /opt/packagename and keep everything together there. The good: everything is in one directory and an uninstall is rm -rf /opt/packagename. The downsides to this are having to add /opt/packagename/bin to everyone's PATH, and the fact that people usually don't put /opt on a separate partition, and you fill up the root partition.

My take on the meanings of the directories (from experience with Debian GNU Linux) is this:

First distinction: s is for superuser

s before a bin directory indicates that it contains tools which are normally only useful to administrators. Note that e.g. ifconfig also belongs to this category and I like to invoke ifconfig as a regular user (to check if I got an IP address / network connectivity) which makes this distinction not as hard as it might seem.

Second distinction: local is for "Local data"

In practice, the most important distinction here is that OS package managers (like APT) install packages to the normal /usr structure and leave /usr/local unaffected. This allows locally compiled packages or company-internal scripts provided by the system administrator to be placed under /usr/local where they will never interfere with the properly packaged files.

It is debatable whether /usr/local should override existing files from the normal /usr structure, see Is it dangerous to have /usr/local/bin ahead of /usr/bin in one's PATH? for some details on this.

Third distinction: The top level /bin and /sbin directories.

On Debian, there is actually a so-called UsrMerge. There used to be the difference that /bin and /sbin were available in the boot process earlier (think of /usr being on a network device or different partition, RAID etc), but nowdays few Linux systems boot well without /usr which is why I would consider the distinction between top-level and /usr directories to be largely of historic interest for Linux.

Finally: The merits and problems of "scattering" files.

There is really no "one true" answer to this. The advantages of Linux' scattered file system are these:

• All binaries reside in few directories. This means you can call every program from the shell. Compare Windows, where one always needs to edit the %PATH% environment variable to add any program of interest. I have seen very long path environments on Windows.

• All libraries reside in a central place. This means they need not be installed twice when used by multiple applications.

• As Linux is designed s.t. most system files are tracked by a package manager anyways, there is no need to keep an overview of the files from a user point of view.

• Due to FHS standardization, documentation also resides in central places allowing systems like man, info and supportive README files to work as expected.

• It is easier to rely on a program being installed in a fixed place. Everyone writes #!/bin/sh -e or similar at the beginning of scripts and it just works. Consider a system where the shell would go to a different directory: How would one know what name it will take? If interested, check the various ways to install Perl or LaTeX on Windows to see how complicated this can get...

The disadvantages I have found so far are these:

• Normally, it is harder to get to run applications "portably" in the sense of "portable applications for Windows". On Linux, it is not as easy to just copy the program to another computer... one needs to have the package (which requires root permissions) or deal with LD_LIBRARY_PATH to point applications to different search paths for the requried libraries.

• Installing multiple versions of the same program needs to be supported explicitly whereas on systems with "one directory per program" this is often less of an issue.

• Managing the files without a package manager is error-prone (as already mentioned).