Log all commands run by admins on production servers

It is company policy for admins to login to the servers via a personal username, and then run sudo -i to become root. Upon running sudo -i, sudo will create an environmental variable called SUDO_USER, which contains the original user's username.

Is there a way to log ALL commands within syslog with something akin to the following syntax:

${TIME/DATE STAMP}: [${REAL_USER}|${SUDO_USER}]: ${CMD}

An example entry would be:

Sat Jan 19 22:28:46 CST 2013: [root|ksoviero]: yum install random-pkg

Obviously it doesn't have to be exactly the above syntax, it just has to include a minimum of the real user (eg. root), the sudo user (eg. ksoviero), and the full command that was run (eg. yum install random-pkg).

I've already tried snoopy, but it did not include the SUDO_USER variable.

Solution 1:

Update: 2 more things that have popped up in the comments and in follow-up questions:

• Using auditd this way will dramatically increase your log volume, especially if the system is heavily in use via commandline. Adjust your log retention policy.
• Auditd logs on the host where they are created are just as secure as other files on the same box. Forward your logs to a remote log collection server like ELK or Graylog to preserve your logs' integrity. Plus, adding to the point above, it allows to more aggressively delete old logs.

As was suggested by Michael Hampton, auditd is the correct tool for the job here.

I tested this on an Ubuntu 12.10 installation, so your mileage may vary on other systems.

• Install auditd:

  apt-get install auditd

• Add these 2 lines to /etc/audit/audit.rules:

  -a exit,always -F arch=b64 -F euid=0 -S execve
  -a exit,always -F arch=b32 -F euid=0 -S execve

These will track all commands run by root (euid=0). Why two rules? The execve syscall must be tracked in both 32 and 64 bit code.

• To get rid of auid=4294967295 messages in logs, add audit=1 to the kernel's cmdline (by editing /etc/default/grub)

• Place the line

  session required pam_loginuid.so

in all PAM config files that are relevant to login (/etc/pam.d/{login,kdm,sshd}), but not in the files that are relevant to su or sudo. This will allow auditd to get the calling user's uid correctly when calling sudo or su.

• Restart your system now.

• Let's login and run some commands:

    $ id -u
    1000
    $ sudo ls /
    bin  boot  data  dev  etc  home  initrd.img  initrd.img.old  lib  lib32  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  scratch  selinux  srv  sys  tmp  usr  var  vmlinuz  vmlinuz.old
    $ sudo su -
    # ls /etc
    [...]

This will yield something like this in /var/log/audit/auditd.log:

----
time->Mon Feb  4 09:57:06 2013
type=PATH msg=audit(1359968226.239:576): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968226.239:576): item=0 name="/bin/ls" inode=2117 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968226.239:576):  cwd="/home/user"
type=EXECVE msg=audit(1359968226.239:576): argc=2 a0="ls" a1="/"
type=SYSCALL msg=audit(1359968226.239:576): arch=c000003e syscall=59 success=yes exit=0 a0=10cfc48 a1=10d07c8 a2=10d5750 a3=7fff2eb2d1f0 items=2 ppid=26569 pid=26570 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/bin/ls" key=(null)
----
time->Mon Feb  4 09:57:06 2013
type=PATH msg=audit(1359968226.231:575): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968226.231:575): item=0 name="/usr/bin/sudo" inode=530900 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968226.231:575):  cwd="/home/user"
type=BPRM_FCAPS msg=audit(1359968226.231:575): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1359968226.231:575): argc=3 a0="sudo" a1="ls" a2="/"
type=SYSCALL msg=audit(1359968226.231:575): arch=c000003e syscall=59 success=yes exit=0 a0=7fff327ecab0 a1=7fd330e1b958 a2=17cc8d0 a3=7fff327ec670 items=2 ppid=3933 pid=26569 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="sudo" exe="/usr/bin/sudo" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.523:578): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.523:578): item=0 name="/bin/su" inode=44 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.523:578):  cwd="/home/user"
type=EXECVE msg=audit(1359968229.523:578): argc=2 a0="su" a1="-"
type=SYSCALL msg=audit(1359968229.523:578): arch=c000003e syscall=59 success=yes exit=0 a0=1ceec48 a1=1cef7c8 a2=1cf4750 a3=7fff083bd920 items=2 ppid=26611 pid=26612 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="su" exe="/bin/su" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.519:577): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.519:577): item=0 name="/usr/bin/sudo" inode=530900 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.519:577):  cwd="/home/user"
type=BPRM_FCAPS msg=audit(1359968229.519:577): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1359968229.519:577): argc=3 a0="sudo" a1="su" a2="-"
type=SYSCALL msg=audit(1359968229.519:577): arch=c000003e syscall=59 success=yes exit=0 a0=7fff327ecab0 a1=7fd330e1b958 a2=17cc8d0 a3=7fff327ec670 items=2 ppid=3933 pid=26611 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="sudo" exe="/usr/bin/sudo" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.543:585): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.543:585): item=0 name="/bin/bash" inode=6941 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.543:585):  cwd="/root"
type=EXECVE msg=audit(1359968229.543:585): argc=1 a0="-su"
type=SYSCALL msg=audit(1359968229.543:585): arch=c000003e syscall=59 success=yes exit=0 a0=13695a0 a1=7fffce08a3e0 a2=135a030 a3=7fffce08c200 items=2 ppid=26612 pid=26622 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="bash" exe="/bin/bash" key=(null)
----
time->Mon Feb  4 09:57:11 2013
type=PATH msg=audit(1359968231.663:594): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968231.663:594): item=0 name="/bin/ls" inode=2117 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968231.663:594):  cwd="/root"
type=EXECVE msg=audit(1359968231.663:594): argc=3 a0="ls" a1="--color=auto" a2="/etc"
type=SYSCALL msg=audit(1359968231.663:594): arch=c000003e syscall=59 success=yes exit=0 a0=7fff8c709950 a1=7f91a12149d8 a2=1194c50 a3=7fff8c709510 items=2 ppid=26622 pid=26661 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/bin/ls" key=(null)

The auid column contains the calling user's uid, which allows you filter for commands run by this user with

 ausearch -ua 1000

This will even list commands the user ran as root.

Sources:

• http://www.woitasen.com.ar/2011/11/auditing-user-actions-after-sudo/
• http://linux.die.net/man/8/pam_loginuid
• http://linux.die.net/man/8/auditd

Solution 2:

Remember that sudo itself logs all sudo commands in the syslog, so all priv'd users should be educated to not simply sudo to get a root shell but to :

sudo command p1 p2 ... pn

The problem with this or any approach I have thought of is that as the root user, it is quite difficult to prevent a user from evading any specific type of logging. Thus anything thing you try will be < 100% I am sorry to say.

Education, documentation, enforcement and above all trust is what is necessary.