logs are not going into log file even after having rsyslog config

I am facing problem. I have two different harpxoy linux servers (in diff network range). In one server, i can see logs are getting generated in log file but on other server, the logs are not getting generated in log file.

I have compared almost all possible config file from two servers and they appear to be same but still cannot find the clue on what is the problem.

Working server -

[root@<hostname> ~]# tail -f  /var/log/haproxy*.log
 
==> /var/log/haproxy.log <==
 
==> /var/log/haproxy-status.log <==
 
==> /var/log/haproxy-traffic.log <==
Oct 28 20:32:48 localhost haproxy[1347]: source_ip:52675 [28/Oct/2021:20:32:46.269] ssl~ logs/es_server1 98/0/1/1391/1950 200 103475 - - --VN 43/0/0/0/0 0/0 "POST /api/telemetry/v2/clusters/_stats HTTP/1.1"
Oct 28 20:33:47 localhost haproxy[1347]: source_ip:52709 [28/Oct/2021:20:33:46.294] ssl~ logs/es_server1 125/0/0/1426/1589 200 103369 - - --VN 43/0/0/0/0 0/0 "POST /api/telemetry/v2/clusters/_stats HTTP/1.1"

ownership of files

-rw------- 1 root root       0 Jan 19  2020 /var/log/haproxy-status.log
-rw------- 1 root root 1980957 Oct 28 20:45 /var/log/haproxy-traffic.log
[root@<hostname> ~]#

ls -l /etc/rsyslog.d/haproxy.conf
-rw-r--r-- 1 root root 265 Jan 14  2020 /etc/rsyslog.d/haproxy.conf

/etc/rsyslog.d/haproxy.conf

# Collect log with UDP
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
 
# Creating separate log files based on the severity
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log
 
# don't put anything in /var/log/messages
& stop

rsyslog.conf file -

[root@<hostname> ~]# cat /etc/rsyslog.conf |grep -v '#'

$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
local6.*,auth.*,authpriv.*      @10.1.x.x

/etc/haproxy/haproxy.cfg file -

global
    user haproxy
    group haproxy
    daemon
    stats socket 127.0.0.1:14567
    log 127.0.0.1:514 local0 debug
 
defaults
    mode http
    log /dev/log local1 notice
 
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option httpclose
    option forwardfor except 127.0.0.0/8
    option redispatch
    option abortonclose
    option httplog
    option dontlognull
    .
    .

both ports mentioned in haproxy config is up.


[root@<hostname> ~]#  netstat -tulnp |grep 514
udp        0      0 127.0.0.1:514           0.0.0.0:*                           1309/rsyslogd
[root@<hostname> ~]#
 
[root@<hostname> ~]#  netstat -antp |grep 14567
tcp        0      0 127.0.0.1:14567         0.0.0.0:*               LISTEN      1347/haproxy
[root@<hostname> ~]#

logs -

journalctl -fu rsyslog

Oct 28 15:00:10 <hostname> rsyslogd[1309]: imjournal: journal reloaded... [v8.24.0-57.el7_9.1 try http://www.rsyslog.com/e/0 ]
Oct 28 20:40:54 <hostname> rsyslogd[1309]: imjournal: journal reloaded... [v8.24.0-57.el7_9.1 try http://www.rsyslog.com/e/0 ]

[root@<hostname> ~]# journalctl -fu haproxy

-- Logs begin at Wed 2021-10-27 02:30:13 +03. --
Oct 27 20:28:29 <hostname> haproxy[1347]: Server tomcat-apps/App_server1 is DOWN, reason: Layer7 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Access logs are also appearing in message log.

cat /var/log/messages | grep haproxy

Oct 28 21:27:47 localhost haproxy[1347]: source_ip:54604 [28/Oct/2021:21:27:46.309] ssl~ logs/es_server1 98/0/1/1364/1550 200 103362 - - --VN 43/0/0/0/0 0/0 "POST /api/telemetry/v2/clusters/_stats HTTP/1.1"
Oct 28 21:28:47 localhost haproxy[1347]: source_ip:54635 [28/Oct/2021:21:28:46.307] ssl~ logs/es_server1 151/0/0/1397/1647 200 103361 - - --VN 43/0/0/0/0 0/0 "POST /api/telemetry/v2/clusters/_stats HTTP/1.1"

Non-working server -

no logs in below files

[root@<hostname> ~]# tail -f  /var/log/haproxy*.log 
==> /var/log/haproxy.log <==
 
==> /var/log/haproxy-status.log <==
 
==> /var/log/haproxy-traffic.log <==

ownership is same as above

[root@<hostname> ~]# ls -l /var/log/haproxy*.log
-rw------- 1 root root 0 Sep  4  2020 /var/log/haproxy.log
-rw------- 1 root root 0 Sep  4  2020 /var/log/haproxy-status.log
-rw------- 1 root root 0 Oct 28 20:23 /var/log/haproxy-traffic.log
[root@<hostname> ~]#

ls -l /etc/rsyslog.d/haproxy.conf
-rw-r--r-- 1 root root 265 Aug  4 20:15 /etc/rsyslog.d/haproxy.conf

/etc/rsyslog.d/haproxy.conf

# Collect log with UDP
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
 
# Creating separate log files based on the severity
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log
 
# don't put anything in /var/log/messages
& stop

cat /etc/rsyslog.conf |grep -v '#'

$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
local6.*    @@10.1.x.x

/etc/haproxy/haproxy.cfg file

global
    user haproxy
    group haproxy
    daemon
    stats socket 127.0.0.1:14567
    log 127.0.0.1:514 local0 debug
 
defaults
    mode http
    log /dev/log local1 notice
 
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option httpclose
    option forwardfor except 127.0.0.0/8
    option redispatch
    option abortonclose
    option httplog
    option dontlognull

ports - here some connection are in TIME_WAIT state for port 14567

[root@<hostname> ~]# netstat -tulnp |grep 514
udp        0      0 127.0.0.1:514           0.0.0.0:*                           21740/rsyslogd
[root@<hostname> ~]#
 
 
[root@<hostname> ~]# netstat -antp |grep 14567
tcp        0      0 127.0.0.1:14567         0.0.0.0:*               LISTEN      18749/haproxy
tcp        0      0 127.0.0.1:14567         127.0.0.1:36168         TIME_WAIT   -
tcp        0      0 127.0.0.1:14567         127.0.0.1:36170         TIME_WAIT   -
tcp        0      0 127.0.0.1:14567         127.0.0.1:36242         TIME_WAIT   -
tcp        0      0 127.0.0.1:14567         127.0.0.1:36348         TIME_WAIT   -
tcp        0      0 127.0.0.1:14567         127.0.0.1:36332         TIME_WAIT   -

Checked the logs but not finding anything related to this issue.

journalctl -fu rsyslog
 
Oct 28 21:00:01 <hostname> rsyslogd[21740]: action 'action 9' resumed (module 'builtin:omfwd') [v8.24.0-57.el7_9.1 try http://www.rsyslog.com/e/2359 ]
Oct 28 21:00:01 <hostname> rsyslogd[21740]: action 'action 9' resumed (module 'builtin:omfwd') [v8.24.0-57.el7_9.1 try http://www.rsyslog.com/e/2359 ]

[root@<hostname> ~]#  journalctl -fu haproxy
-- Logs begin at Thu 2021-10-28 10:23:25 +03. --
Oct 28 16:34:17 <hostname> haproxy[11346]: Proxy stats started.
Oct 28 19:59:29 <hostname> systemd[1]: Stopping HAProxy Load Balancer...
Oct 28 19:59:29 <hostname> haproxy-systemd-wrapper[11345]: haproxy-systemd-wrapper: SIGTERM -> 11347.
Oct 28 19:59:29 <hostname> haproxy[18748]: Proxy ssl started.
Oct 28 19:59:29 <hostname> haproxy[18748]: Proxy stats started.

here message log file has haproxy access log but its not going to its individual file.

cat /var/log/messages | grep  haproxy
 
Oct 28 10:19:52 <hostname> audispd: node=<hostname> type=SERVICE_START msg=audit(1635405592.054:45): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=haproxy comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 28 10:19:52 <hostname> haproxy[1749]: Proxy ssl-pingaccess started.
Oct 28 10:19:52 <hostname> haproxy[1749]: Proxy ssl started.
 
Oct 28 10:19:52 <hostname> haproxy[1749]: Server ssl-pingaccess/192.168.157.41 is DOWN, reason: Layer4 connection problem, info: "Connection error during SSL handshake (Connection refused) at initial connection step of tcp-check", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 28 10:19:52 <hostname> haproxy[1749]: backend ssl-pingaccess has no server available!

These files are same on both servers

both are same
 
/usr/lib/systemd/system/rsyslog.service
 
/usr/lib/systemd/system/haproxy.service
 
/etc/sysconfig/haproxy
 
/etc/sysconfig/rsyslog
 
/etc/logrotate.d/haproxy

Both have rsyslog-8.24.0-57.el7_9.1.x86_64 and haproxy-1.5.18-9.el7_9.1.x86_64 package.

Configuration is almost same on both the servers but on non-working servers, logs are not going to its own log file.

Can anyone please help.

Thanks,

Solution 1:

After adding below line under frontend section, logs are now generating in log file.

log 127.0.0.1:514 local0 debug

logs are not going into log file even after having rsyslog config

Solution 1:

Related

Recent Posts