Ubuntu 18.04 server - how to change or adjust operating system logging for a certain systemd service?
I've tried quite a bit of Googling on this but have not come up with much so I figured I'd ask here. Here's my current problem:
I'm running Ubuntu 18.04 server pretty much stock. The purpose of the server is to run a vision app. The problem I'm having is when I check the journalctl for the vision app service, PAM (Pluggable Authentication Module) and some other operating system related services are logging excessively to the vision app, which I would rather they not do. Here is an example:
journalctl -u visionapp.service | less
output:
(some stuff omitted)
Jan 08 10:43:12 visionapp sudo[2483]: root : TTY=unknown ; PWD=/opt ; USER=root ; COMMAND=/sbin/iptables -F
Jan 08 10:43:12 visionapp sudo[2483]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 08 10:43:12 visionapp sudo[2483]: pam_unix(sudo:session): session closed for user root
Jan 08 10:43:12 visionapp sudo[2490]: root : TTY=unknown ; PWD=/opt ; USER=root ; COMMAND=/sbin/sysctl net.ipv4.conf.xxx999.forwarding=1
Jan 08 10:43:12 visionapp sudo[2490]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 08 10:43:12 visionapp VisionApp[2471]: net.ipv4.conf.xxx999.forwarding = 1
Jan 08 10:43:12 visionapp sudo[2490]: pam_unix(sudo:session): session closed for user root
Jan 08 10:43:12 visionapp sudo[2493]: root : TTY=unknown ; PWD=/opt ; USER=root ; COMMAND=/sbin/sysctl net.ipv4.conf.yyy888.forwarding=1
Jan 08 10:43:12 visionapp sudo[2493]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 08 10:43:13 visionapp VisionApp[2471]: net.ipv4.conf.yyy888.forwarding = 1
Jan 08 10:43:13 visionapp sudo[2493]: pam_unix(sudo:session): session closed for user root
Jan 08 10:43:13 visionapp VisionApp[2471]: route: sudo iptables -t nat -A PREROUTING -p tcp -d 88.88.88.63 --dport 77777 -j DNAT --to-destination 99.99.99.35:77777
Jan 08 10:43:13 visionapp sudo[2496]: root : TTY=unknown ; PWD=/opt ; USER=root ; COMMAND=/sbin/iptables -t nat -A PREROUTING -p tcp -d 88.88.88.63 --dport 77777 -j DNAT --to-destination 99.99.99.35:77777
Jan 08 10:43:13 visionapp sudo[2496]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 08 10:43:13 visionapp sudo[2496]: pam_unix(sudo:session): session closed for user root
Jan 08 10:43:13 visionapp VisionApp[2471]: route: sudo iptables -t nat -A PREROUTING -p tcp -d 88.88.88.63 --dport 5555 -j DNAT --to-destination 99.99.99.11:80
Jan 08 10:43:13 visionapp sudo[2499]: root : TTY=unknown ; PWD=/opt ; USER=root ; COMMAND=/sbin/iptables -t nat -A PREROUTING -p tcp -d 88.88.88.63 --dport 5555 -j DNAT --to-destination 99.99.99.11:80
Jan 08 10:43:13 visionapp sudo[2499]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 08 10:43:13 visionapp sudo[2499]: pam_unix(sudo:session): session closed for user root
Jan 08 10:43:13 visionapp VisionApp[2471]: route: sudo iptables -t nat -A PREROUTING -p tcp -d 88.88.88.63 --dport 6666 -j DNAT --to-destination 99.99.99.30:80
Jan 08 10:43:13 visionapp sudo[2502]: root : TTY=unknown ; PWD=/opt ; USER=root ; COMMAND=/sbin/iptables -t nat -A PREROUTING -p tcp -d 88.88.88.63 --dport 6666 -j DNAT --to-destination 99.99.99.30:80
Jan 08 10:43:13 visionapp sudo[2502]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 08 10:43:13 visionapp sudo[2502]: pam_unix(sudo:session): session closed for user root
Jan 08 10:43:13 visionapp VisionApp[2471]: route: sudo iptables -t nat -A PREROUTING -p udp -d 88.88.88.63 --dport 23456 -j DNAT --to-destination 99.99.99.30:23456
Jan 08 10:43:13 visionapp sudo[2505]: root : TTY=unknown ; PWD=/opt ; USER=root ; COMMAND=/sbin/iptables -t nat -A PREROUTING -p udp -d 88.88.88.63 --dport 23456 -j DNAT --to-destination 99.99.99.30:23456
Jan 08 10:43:13 visionapp sudo[2505]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 08 10:43:13 visionapp sudo[2505]: pam_unix(sudo:session): session closed for user root
Jan 08 10:43:13 visionapp VisionApp[2471]: route: sudo iptables -t nat -A PREROUTING -p tcp -d 88.88.88.63 --dport 28208 -j DNAT --to-destination 99.99.99.30:28208
Jan 08 10:43:13 visionapp sudo[2508]: root : TTY=unknown ; PWD=/opt ; USER=root ; COMMAND=/sbin/iptables -t nat -A PREROUTING -p tcp -d 88.88.88.63 --dport 28208 -j DNAT --to-destination 99.99.99.30:28208
Jan 08 10:43:13 visionapp sudo[2508]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 08 10:43:13 visionapp sudo[2508]: pam_unix(sudo:session): session closed for user root
Jan 08 10:43:13 visionapp VisionApp[2471]: route: sudo iptables -t nat -A PREROUTING -p tcp -d 88.88.88.63 --dport 2112 -j DNAT --to-destination 99.99.99.36:2112
Jan 08 10:43:13 visionapp sudo[2511]: root : TTY=unknown ; PWD=/opt ; USER=root ; COMMAND=/sbin/iptables -t nat -A PREROUTING -p tcp -d 88.88.88.63 --dport 2112 -j DNAT --to-destination 99.99.99.36:2112
Jan 08 10:43:13 visionapp sudo[2511]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 08 10:43:13 visionapp sudo[2511]: pam_unix(sudo:session): session closed for user root
Jan 08 10:43:13 visionapp VisionApp[2471]: Make Routable: sudo iptables -t nat -A POSTROUTING -j MASQUERADE
Jan 08 10:43:13 visionapp sudo[2514]: root : TTY=unknown ; PWD=/opt ; USER=root ; COMMAND=/sbin/iptables -t nat -A POSTROUTING -j MASQUERADE
Jan 08 10:43:13 visionapp sudo[2514]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 08 10:43:13 visionapp sudo[2514]: pam_unix(sudo:session): session closed for user root
(some stuff omitted)
I changed some of the names and numbers to protect company anonymity but otherwise this is actual output. I'd prefer to be able to suppress this in some way.
After consulting this post https://unix.stackexchange.com/questions/327301/how-to-stop-sudo-pam-messages-in-auth-log-for-a-specific-user-on-ubuntu-16-04 I'd prefer to stay away from the PAM config files for 3 reasons:
1) I tried what the post suggested and it didn't work, and caused the vision app to crash.
2) Making a mistake editing the PAM logs could cause a lockout of root access
3) Some of the above messages do not seem to be generated by PAM
The last answer in the above post mentions filtering at the syslog level. I've tried to read up on this but haven't been able to work out much so far. I've at least been able to determine that the critical files seem to be /etc/rsyslog.conf and the files in
/etc/rsyslog.d/.
here is my /etc/rsyslog.conf/:
$ cat /etc/rsyslog.conf
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
There are 3 files in rsyslog.d:
$ cd /etc/rsyslog.d
$ ls -l
-rw-r--r-- 1 root root 314 Aug 15 2017 20-ufw.conf
-rw-r--r-- 1 root root 255 Apr 27 2018 21-cloudinit.conf
-rw-r--r-- 1 root root 1124 Jan 30 2018 50-default.conf
I'm under the impression that 20-ufw.conf and 21-cloudinit.conf are for some other specific purposes. Here is 50-default.conf:
$ cat 50-default.conf
# Default rules for rsyslog.
#
# For more information see rsyslog.conf(5) and /etc/rsyslog.conf
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
#user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info -/var/log/mail.info
#mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Some "catch-all" log files.
#
#*.=debug;\
# auth,authpriv.none;\
# news.none;mail.none -/var/log/debug
#*.=info;*.=notice;*.=warn;\
# auth,authpriv.none;\
# cron,daemon.none;\
# mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
To my knowledge these files are stock for an Ubuntu 18.04 server install.
So here are my questions at this point:
1) Should I edit one of the above files, or create another file in /etc/rsyslog.d, ex. 20-visionapp.conf or similar?
2) Is there a way to change the above files to conditionally not include log messages for visionapp.service? i.e. if a log line contains pam_unix(sudo:session) or root : TTY=unknown to not include it? If somebody can suggest such a line, please clarify if it would apply to all systemd services or only to visionapp.service specifically, and also if it would apply to all users or for a specific user? If options are available where both of these could be chosen that would be even better.
-- Update --
After more Googling, I did the following:
cd /etc/rsyslog.d
sudo nano 20-visionapp.conf
in nano I entered:
:msg,contains,"pam_unix" /var/log/PAM.log
& stop
then from the command line again I did:
service rsyslog restart
Then started and stopped the vision app again. I was hoping that any message containing pam_unix would now go to the file /var/log/PAM.log, but when I ran journalctl -u visionapp.service | less the pam_unix messages were still there.
I think I'm at least getting close here. What am I doing wrong? Any suggestions?
-- Update2 --
Based on this documentation https://www.rsyslog.com/discarding-unwanted-messages/ for /etc/rsyslog.d/20-visionapp.conf, I also tried:
:msg,contains,"pam_unix" ~
and
:msg, contains, "pam_unix" ~
Neither of these work, i.e. journalctl -u visionapp.service | less still shows the pam_unix messages.
I should also mention that this post https://unix.stackexchange.com/questions/133898/why-does-rsyslogd-not-honor-the-following-lines-in-rsyslog-d describes a very similar problem and does not yet have an accepted answer.
-- Update3 --
If I do this:
sudo nano /etc/rsyslog.d/19-visionapp.conf
then enter:
:msg, contains, "pam_unix" /var/log/visionapp-other.log
& stop
Then all the messages with pam_unix log to both /var/log/visionapp-other.log and to journalctl -u visionapp.service | less. It seems from this post https://unix.stackexchange.com/questions/8737/rsyslog-is-not-discarding-message-as-it-should this has been a known bug in the past. Does anybody have a workaround or any more information on this?
-- Update4 --
After more Googling I'm convinced that the steps I mentioned in the previous update are correct and there is either a bug in rsyslog or the integration of rsyslog into Ubuntu 18.04 server. The workaround I've settled on for the moment is to make a script in the home directory containing:
journalctl -u visionapp.service | grep -v "pam_unix" | grep -v "TTY=unknown" | less
This screens out the messages with pam_unix and TTY=unknown that I'd rather not see. This is clearly not a great solution and I'm disappointed in rsyslog and Ubuntu for not providing a better way to modify systemd log output.
Solution 1:
you can do this by editing /etc/syslog.conf
like this:
*.=info;*.=notice;*.=warning;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
you can change =warning to =notice =info etc according to the logging level you want