How to ensure user separation on an old school shell server

Solution 1:

hidepid

procfs on Linux now supports the hidepid option. From man 5 proc:

hidepid=n (since Linux 3.3)
      This   option   controls  who  can  access  the  information  in
      /proc/[pid]  directories.   The  argument,  n,  is  one  of  the
      following values:

      0   Everybody  may  access all /proc/[pid] directories.  This is
          the traditional behavior, and  the  default  if  this  mount
          option is not specified.

      1   Users  may  not  access  files and subdirectories inside any
          /proc/[pid]  directories  but  their  own  (the  /proc/[pid]
          directories  themselves  remain  visible).   Sensitive files
          such as /proc/[pid]/cmdline and /proc/[pid]/status  are  now
          protected  against other users.  This makes it impossible to
          learn whether any user is running  a  specific  program  (so
          long  as  the program doesn't otherwise reveal itself by its
          behavior).

      2   As for mode 1, but in addition the  /proc/[pid]  directories
          belonging  to other users become invisible.  This means that
          /proc/[pid] entries can no longer be used  to  discover  the
          PIDs  on  the  system.   This  doesn't  hide the fact that a
          process with a specific PID value exists (it can be  learned
          by  other  means,  for  example,  by "kill -0 $PID"), but it
          hides a process's UID and  GID,  which  could  otherwise  be
          learned  by  employing  stat(2)  on a /proc/[pid] directory.
          This greatly complicates an  attacker's  task  of  gathering
          information   about  running  processes  (e.g.,  discovering
          whether some daemon is  running  with  elevated  privileges,
          whether  another  user  is  running  some sensitive program,
          whether other users are running any program at all,  and  so
          on).

gid=gid (since Linux 3.3)
      Specifies  the  ID  of  a  group whose members are authorized to
      learn  process  information  otherwise  prohibited  by   hidepid
      (ie/e/,  users  in this group behave as though /proc was mounted
      with hidepid=0.  This group should be used instead of approaches
      such as putting nonroot users into the sudoers(5) file.

So, mounting /proc with hidepid=2 is enough to hide the details of other users' processes on Linux > 3.3. Ubuntu 12.04 comes with 3.2 by default, but you can install newer kernels. Ubuntu 14.04 and above easily match this requirement.

ACLs

As a first step, remove rwx permissions for others from every home directory (and for group as well, if you require it). I'm assuming, of course, that the folder(s) containing the home directories don't have write permissions to anybody except root.

Then, grant services like the web server and mail server access to the appropriate directories using ACLs. For example, to grant the web server process access to the user home pages, assuming www-data is the user and ~/public_html is where the home page is kept:

setfacl u:www-data:X ~user
setfacl d:u:www-data:rX ~user/public_html

Similarly, add ACLs for the mail processes and the mailbox directories.

ACLs are enabled by default on ext4 at least on Ubuntu 14.04 and above.

/tmp and umask

Another problem is /tmp. Set the umask so that files aren't group- or world-readable, so that users' temporary files aren't accessible to other users.

With these three settings, users shouldn't be able to access other users' files, or examine their processes.