Outputting JSON logs on Elastic Beanstalk with Amazon Linux 2

Solution 1:

I found a solution that works well enough, so I'll post it here for posterity. If someone can suggest a better one, please do.

Elastic Beanstalk on Amazon Linux 2 relies on rsyslog for log processing and output. There's a file at /opt/elasticbeanstalk/config/private/rsyslog.conf that during deployment gets copied over to /etc/rsyslog.d/web.conf and it's the one that's directing all output from the web application to /var/log/web.stdout.log.

The file doesn't include any custom template. It relies on rsyslog's default template, which prefixes any %msg% with a timestamp and $programname (which is web in this case).

I tried replacing this file via an .ebextensions config, but that didn't work, because Elastic Beanstalk seems to be overwriting this file after .ebextensions run. So I added an additional platform hook that deletes the file, keeping the custom one I added.

Here's the .ebextensions/logs.config file:

files:
  "/etc/rsyslog.d/web-files.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      template(name="WebTemplate" type="string" string="%msg%\n")

      if $programname == 'web' then {
        *.=warning;*.=err;*.=crit;*.=alert;*.=emerg; /var/log/web.stderr.log;WebTemplate
        *.=info;*.=notice /var/log/web.stdout.log;WebTemplate
      }

commands:
  remove-.bak-rsyslog:
    command: rm -f *.bak
    cwd: /etc/rsyslog.d

and .platform/hooks/predeploy/remove-default-rsyslog-conf.sh (make sure you chmod +x this one):

#!/bin/sh
rm /etc/rsyslog.d/web.conf
systemctl restart rsyslog.service

Solution 2:

I used Platform Hooks to accomplish this. The only catch is /etc/rsyslog.d/web.conf is replaced on both application and configuration deployments, so you need a hook for both.

This approach avoids messing with Elastic Beanstalk's internal files in /opt/elasticbeanstalk/config/private (which have changed since the previous answers - rsyslog.conf no longer exists). Also, platform hooks are now preferred over ebextensions.

If you use CodeBuild, don't forget to include the platformFiles directory (or wherever you put your files) in your output artifact.

NOTE: This code assumes the name of the process is web. If you defined a different process name in your Procfile, use that instead. However, I think the rsyslog config should always be at /etc/rsyslog.d/web.conf despite the process name.

Make sure all of your .sh files are executable using chmod +x.

.platform/hooks/predeploy/10_logs.sh

#!/bin/sh
sudo platformFiles/setupLogs.sh

.platform/confighooks/predeploy/10_logs.sh

#!/bin/sh
sudo platformFiles/setupLogs.sh

platformFiles/setupLogs.sh

#!/bin/sh
# By default logs output to /var/log/web.stdout.log are prefixed. We want just the raw logs from the app.
# This updates the rsyslog config. Also grants read permissions to the log files.

set -eu

mv platformFiles/rsyslogWebConf.conf /etc/rsyslog.d/web.conf

touch /var/log/web.stdout.log
touch /var/log/web.stderr.log
chmod +r /var/log/web.stdout.log
chmod +r /var/log/web.stderr.log

systemctl restart rsyslog.service

platformFiles/rsyslogWebConf.conf

# This file is created from Elastic Beanstalk platform hooks.

template(name="WebTemplate" type="string" string="%msg%\n")

if $programname == 'web' then {
  *.=warning;*.=err;*.=crit;*.=alert;*.=emerg; /var/log/web.stderr.log;WebTemplate
  *.=info;*.=notice /var/log/web.stdout.log;WebTemplate
}

Conjecture

It looks like /opt/elasticbeanstalk/config/private/rsyslog.conf was replaced with /opt/elasticbeanstalk/config/private/rsyslog.conf.template:

# This rsyslog file redirects Elastic Beanstalk platform logs.
# Logs are initially sent to syslog, but we also want to divide
# stdout and stderr into separate log files.

{{range .ProcessNames}}if $programname  == '{{.}}' then {
  *.=warning;*.=err;*.=crit;*.=alert;*.=emerg /var/log/{{.}}.stderr.log
  *.=info;*.=notice /var/log/{{.}}.stdout.log
}
{{end}}

Based on this, I speculate Elastic Beanstalk uses this template to generate a single /etc/rsyslog.d/web.conf file which contains a block for every defined process name. Because both an application and a configuration deployment can change the defined processes, it makes sense this file is recreated after both.