How to automate certbot certificate renewal on Ubuntu 20.04
I'm running certbot on Ubuntu 20.04 in AWS, installed as a snap package. I'm not sure if certbot renewal is running properly. I'd appreciate some help working out how to best get it working.
This is a new server, which I turn on and off while I'm getting it ready for production. It runs about 8 - 10 hours a day at the moment. It's not often running at midnight, which I think is when the cron job runs. It will be on 24/7 in a few days once I finish the configuration.
One thing I found is this question answer saying
You shouldn't have to set up anything. Any recent Debian/Ubuntu install of certbot should install a systemd timer and a cron job (and the cron job will only run certbot if systemd is not active, so you don't get both running).
It looks to me like the certbot timer isn't running, and if it did it appears to be pointing at /dev/null. Because systemd is active I wonder if the cron job is running.
Timers and systemd
I found a comment there may be an issue with timers and snap so maybe this is a known issue.
systemctl list-timers
The timer doesn't seem to run
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2021-03-17 23:44:00 UTC 3h 24min left n/a n/a snap.certbot.renew.timer snap.certbot.renew.service
Certbot timer appears to point at /dev/null. This question indicates that's not how it should be.
> root@aws2:/etc/systemd/system# ls -l | grep certbot
lrwxrwxrwx 1 root root 9 Jan 9 06:38 certbot.timer -> /dev/null
I can see the following in syslog but I'm not sure what it means
Mar 17 16:51:02 aws2 systemd[1]: Started Timer renew for snap application certbot.renew.
Cron
In syslog I can see this the cron job is running but there's no output
Mar 16 00:00:01 aws2 CRON[2072]: (root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)
The following is in /etc/cron.d/certbot (this was presumably put there by the certbot installation, I get the general idea of what it does but I don't know what the test / perl stuff does)
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew
When I run the whole command (root test /x etc) I get the message below.
Command 'root' not found, but can be installed with: snap install root-framework
When I run this part I get no output (note I have removed the "-q" from certbot for testing). I'm not sure what the test part is doing, but certbot doesn't seem to do anything when I run this command.
> test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot renew
Key questions
- Any ideas what's up with the systemd timer, why it's pointing at /dev/null? Or should I just ignore this known issue?
- Should I "snap install root-framework" to install "root" like Ubuntu is suggesting?
- The cron job "root test" doesn't appear to be doing anything... any anyone explain what is being tested there and whether "cerbot renew" it's actually running?
Update - Proposed Solution
In the /etc/cron.daily folder I've created the following folder. I think it will do what I want, I'll check logs at some point to see. I'm still interested in the questions I asked above.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
/usr/bin/certbot -q renew
Solution 1:
Since you're using snap, it's worth double checking to see if there's a second timer in place (other than certbot-renew) which is implemented by snap, and does do the renewal properly.
Here you can see I have both a mask to make sure that the normal certbot.timer is disabled, but I also have a timer installed by snap:
root@pi:/etc/systemd/system # ls -l *certbot*
lrwxrwxrwx 1 root root 9 Oct 6 10:42 certbot.timer -> /dev/null
-rw-r--r-- 1 root root 293 Oct 6 05:26 snap.certbot.renew.timer
and I can see that this runs twice daily and is currently waiting:
root@pi:/etc/systemd/system # systemctl list-units | grep certbot.renew
snap.certbot.renew.timer loaded active waiting Timer renew for snap application certbot.renew
root@pi:/etc/systemd/system # cat snap.certbot.renew.timer [Unit]
# Auto-generated, DO NOT EDIT
Description=Timer renew for snap application certbot.renew
Requires=snap-certbot-1515.mount
After=snap-certbot-1515.mount
X-Snappy=yes
[Timer]
Unit=snap.certbot.renew.service
OnCalendar=*-*-* 07:09
OnCalendar=*-*-* 19:46
[Install]
WantedBy=timers.target
Don't be fooled by the cronjob. It's double checking to make sure that it's not doing something that's already managed by systemd:
# grep test /etc/cron.d/certbot
0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew
the test used in there returns a '1' code, and hence anything after the && isn't executed.
Your workaround won't cause issues - but the stock installation should do the right thing.
So to answer your three questions::
Any ideas what's up with the systemd timer, why it's pointing at /dev/null? Or should I just ignore this known issue?
This is double checking that the snap timer is used and any system timer (e.g. when installed via apt) for certbot isn't
Should I "snap install root-framework" to install "root" like Ubuntu is suggesting?
Cron files stored in in /etc/cron.d have a syntax that include the username to be used when executing. root here is indicating that the cronjob is run as the root user. You don't need to install root-framework - the word 'root' isn't part of the command itself.
The cron job "root test" doesn't appear to be doing anything... any anyone explain what is being tested there and whether "cerbot renew" it's actually running?
As above - this cronjob is validly saying 'Hey, systemd is running! I won't do anything!'
Finally, you can verify the status of your renewals in your logfile:
# grep -c skipped /var/log/letsencrypt/letsencrypt.log
14
In my case, no renewals were happening yet (validity is long enough) but it was running - just very quietly.