Why is this rsync + ssh cron job giving me 'Permission denied (publickey)' errors?

I make frequent backups to a local drive which I want to sync daily to a remote server.

The target server is configured for SSH key (no password) access only. Since my primary SSH key for that server is passphrase-protected, I've created a second SSH key (not passphrase protected) + user to use for unattended backups - this way I do not have to be present to enter my passphrase when cron runs.

I'm using cron and rsync, and all of the commands work individually, but fail when combined.

The furthest I've got while troubleshooting is running

env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/"

which returns the error

Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]

Any tips on how to troubleshoot this further?

Here's what I've tried so far and I'm out of ideas:

1. Cron is definitely running ps aux | grep cron

2. Nothing unusual in /var/log/syslog Sep 7 13:22:01 desktop CRON[6735]: (tom) CMD (sh /home/tom/Documents/Scripts/offsite-backup)

3. SSH in Terminal to remote server as the backup user works ssh [email protected]

4. Running the command in Terminal works perfectly rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/

5. Manually specifying the path to the backups-user key has no effect rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/

6. Replacing the non-functioning command with a simple test command works echo "Hello world" > ~/Desktop/test.txt

7. Shouting/swearing at the computer had no effect (but made me feel better temporarily).

Edit 1:

Here's my crontab file and the script it calls.

...
# m h  dom mon dow   command
MAILTO=""
* * * * * sh /home/tom/Documents/Scripts/offsite-backup

and

#!/bin/bash

rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/

Edit 2:

Just to clarify, /var/log/auth.log on the target server contains the line Sep 11 08:23:01 <hostname> CRON[24421]: pam_unix(cron:session): session closed for user root This is confusing because I'm no longer running cron every minute locally, but a new entry still appears every minute in the server logs. Crontab files for all users (including root) on the server are empty & do nothing.

Also, user 'backups-only' was created only on the server and with limited rights, with a dedicated SSH key copied to my desktop machine. I'm assuming this is the way to go because everything works when running the commands manually.

The crontab file posted above is for me, user 'tom' on my desktop machine. My intent is to have it call the script which should log in to the server as user 'backups-only'. I just tried running the backup script (rather than the command inside it) and it successfully connected & worked. I ran it on my desktop as user 'tom', same user who created the cron job that won't work. Here's the output from the server log corresponding with that successful login

Sep 11 08:35:31 <hostname> sshd[25071]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Sep 11 08:35:32 <hostname> sshd[25071]: Accepted publickey for backups-only from <desktop IP> port 54242 ssh2: RSA e2:e6:07:27:c1:continues...
Sep 11 08:35:32 <hostname> sshd[25071]: pam_unix(sshd:session): session opened for user backups-only by (uid=0)
Sep 11 08:35:32 <hostname> systemd-logind[638]: New session 12 of user backups-only.
Sep 11 08:36:00 <hostname> sshd[25133]: Received disconnect from <desktop IP>: 11: disconnected by user
Sep 11 08:36:00 <hostname> sshd[25071]: pam_unix(sshd:session): session closed for user backups-only

Solution 1:

Since everything is working fine from the command line, the error Permission denied (publickey) means that the SSH part of rsync is using a different identity file than the specified username.

From Jan's comment on the original question, we can specify the identity file in the rsync command using -e 'ssh -i /path/to/identity.file' ....

Using the below command to start off with a fresh environment in cron and specifying the complete path to the file apparently solves the issue:

env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/"

_{I'm still really interested in this finding. It probably has to do with cron, the fact that it starts with minimal environment variables, and the ssh-agent. I'll be setting up the same scenario ina a couple of days to test it out and report back.}

Solution 2:

I just have solved this problem that has kept me busy ..

Unable to connect in RSYNC over SSH, despite having stipulated the identity for SSH ...Nothing is done ... Rsync says "permission denied" and ssh tells me "read_passphrase: can not open /dev/tty: No device or address of this type"

But I read a post that explained that the crontab has its own environment that is not the same as root. I already knew that but I did not understand the impact it could have on SSH when using the SSH-AGENT

But my SSH key exchanges are done with PassPhrase ... so if the environment is different and my RSYNC over SSH expects a passphrase that can not be entered => SSH debug info also indicate the error:

"debug1: read_passphrase: can not open /dev/tty: No such device or address" => Well yes no TTY = no passphrase = not allowed

On my machine I use "Keychain" to have the SSH agent launched so I do not have to reenter the passphrase every time I try a remote connection. Keychain generates a file that contains the following information

SSH_AUTH_SOCK = /tmp/ssh-PWg3yHAARGmP/agent.18891; export SSH_AUTH_SOCK; SSH_AGENT_PID = 18893; export SSH_AGENT_PID;

==> The SSH-AGENT command returns the same info.

So, in the end, it is these information related to the current session that allow future authentications of the current session, without the need to enter the passphrase because already done previously and memorized ...

==> The solution is there ... it is enough in the script launched by the crontab, and to "source" the file containing this information or to do it on the command line ds the crontab ...

example: 14 09 * * *. /home/foo/.keychain/foo.serveur.org-sh && scp -vvv -P 22 /tmp/mon_fic/toto.sh [email protected] :. >> / var / log / check_connexion.log 2> & 1 or use the command "source /home/foo/keychain/foo.server.org-sh" in the script that starts a connection using SSH.

=> With this sourcing, no more worry. The information of SSH_AUTH_SOCK and SSH_AGENT_PID are loaded in the environment of the Crontab and are therefore known, the RSYNC over SSH works without any problem.

It has kept me busy but now, it works :)

Solution 3:

Caveat for those who use SSH Agent Forwarding:

If you see this behaviour when debugging a script on a remote host, it's because even with the -e "ssh -i /path/to/key" flag, ssh will use your local (forwarded) key rather than the one on the server.

Concrete example: I have a script on the dev server that pulls in data from the "data server" using rsync over ssh. When I log into the dev server and run it, all is well, but when running from cron, I get the permission denied. Adding some verbosity to the SSH process (flag -vv) I noticed the following:

debug2: key: /home/nighty/.ssh/id_rsa (0x562d8b974820),
debug2: key: /home/juanr/.ssh/id_rsa (0x562d8b962930), explicit
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/nighty/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp 1a:19:08:9f:80:16:b1:db:55:42:9a:52:b2:49:9b:0a
debug1: Authentication succeeded (publickey).

What clued me off here is that by pure chance, I happen to have a different username on the local host ("nighty") than on the dev server ("juanr").

Notice how it marks the key on the dev server as "explicit", but still uses the forwarded key from my laptop to log in. Doing an ssh-copy-id at this point resolves nothing, because it simply reinstalls the forwarded key rather than the one from the dev server. If you use ssh-copy-id with agent forwarding, you need to specify which key to install with the -i flag: ssh-copy-id -i ~/.ssh/id_rsa.pub user@host.