Endless ssh attempt from my IP using my public key ? Fail2ban fail to log anything useful

Lately, my hostinger server has been acting strange, I have hard time accessing the site hosted on it, I can't even ssh, however when I restart the server from hostinger panel. It works again.

Upon checking top, there was sshd process hogging 100% cpu, then I decided to view /var/log/auth.log.

Above gif image is my output of sudo tail -f /var/log/auth.log, up until now it still run endlessly without any break, the output appear to be accepted login log from my IP— and then disconnect (as in disconnect from ssh), and then reconnect again, and disconnect. I've been sitting here for 30 minute and it still spouting same thing.

Any clue of what might this be ? Is this actually an breach attempt ? I have setup fail2ban and ufw on this server, fail2ban logs doesn't yield any strange line.

top yield this.

By default sshd doesn't log wrong attempts of public keys (you'd need to increase its log level to enable them (it requires LogLevel VERBOSE in sshd_config)...

Or you can simply switch to aggressive mode in an jail of fail2ban - set the mode for sshd jail in your jail.local:

[sshd]
mode = aggressive
enabled = true

Hereafter fail2ban would also consider not authenticated attempts (bots, port-scanner, etc connecting to sshd-listener) and ban such evildoers.

To see the difference you can compare the output of these 2 commands:

?sudo? fail2ban-regex -v /var/log/auth.log sshd
?sudo? fail2ban-regex -v /var/log/auth.log 'sshd[mode=aggressive]'

After a week of this mess, I managed to catch the culprit, it is actually my Pi's who caused this whole thing to begin with— that also explain why auth.log showing login from my IP, and to top it off, it is using public key authentication (I only allow pubkey auth for my trusted device).

How did I find out about this ? The comment from @Kamil Maciorowski shed some light to my finding, so I try turning off the Pi and then my server would ran smoothly again— with no bizarre cpu usage.

I checked on my Pi and found the offending script, specifically following line.

rsync --remove-source-files -e "ssh -o ServerAliveInterval=60" -chaP --stats --info=progress2 $host:/home/user/mybackup/dbonly/$(date +"%Y-%m-%d") $backup_location/$host

As you can see I'm using $(date +"%Y-%m-%d") which return today date in yyyy-mm-dd format, the problem is I have append hh:mm to my backup directory— BUT have not changed the script to handle new naming format.

_{I changed the naming at November 12th}

My suspicions were confirmed after I take a look at my rsync log, it clearly shows that it tried to rsync a nonexistant directory— over and over.

sent 8 bytes  received 115 bytes  14.47 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [Receiver=3.1.1]
rsync: [Receiver] write error: Broken pipe (32)
+ [[ 5 -lt 5 ]]
++ date '+%Y-%m-%d %H:%M'
+ fail '2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.'
+ echo 2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.
2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.
+ true
+ rsync --remove-source-files -e 'ssh -o ServerAliveInterval=60' -chaP --stats --info=progress2 vps-strat:/home/user/mybackup/dbonly/2021-11-13 /media/pi/local_backup/db-server/vps-strat
receiving incremental file list
rsync: link_stat "/home/user/mybackup/dbonly/2021-11-13" failed: No such file or directory (2)

Number of files: 0
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 115
Total bytes sent: 8
Total bytes received: 115

sent 8 bytes  received 115 bytes  12.95 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [Receiver=3.1.1]
rsync: [Receiver] write error: Broken pipe (32)
+ [[ 5 -lt 5 ]]
++ date '+%Y-%m-%d %H:%M'
+ fail '2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.'
+ echo 2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.
2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.
+ true
+ rsync --remove-source-files -e 'ssh -o ServerAliveInterval=60' -chaP --stats --info=progress2 vps-strat:/home/user/mybackup/dbonly/2021-11-13 /media/pi/local_backup/db-server/vps-strat
receiving incremental file list
rsync: link_stat "/home/user/mybackup/dbonly/2021-11-13" failed: No such file or directory (2)

Number of files: 0
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 115
Total bytes sent: 8
Total bytes received: 115

sent 8 bytes  received 115 bytes  16.40 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [Receiver=3.1.1]
rsync: [Receiver] write error: Broken pipe (32)
+ [[ 5 -lt 5 ]]
++ date '+%Y-%m-%d %H:%M'
+ fail '2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.'
+ echo 2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.
2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.
+ true
+ rsync --remove-source-files -e 'ssh -o ServerAliveInterval=60' -chaP --stats --info=progress2 vps-strat:/home/user/mybackup/dbonly/2021-11-13 /media/pi/local_backup/db-server/vps-strat
receiving incremental file list
rsync: link_stat "/home/user/mybackup/dbonly/2021-11-13" failed: No such file or directory (2)

Number of files: 0
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 115
Total bytes sent: 8
Total bytes received: 115

sent 8 bytes  received 115 bytes  12.95 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [Receiver=3.1.1]
rsync: [Receiver] write error: Broken pipe (32)
+ [[ 5 -lt 5 ]]
++ date '+%Y-%m-%d %H:%M'
+ fail '2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.'
+ echo 2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.
2021-11-13 03:43 // The command has failed after 5 attempts for vps-strat.
+ true
+ rsync --remove-source-files -e 'ssh -o ServerAliveInterval=60' -chaP --stats --info=progress2 vps-strat:/home/user/mybackup/dbonly/2021-11-13 /media/pi/local_backup/db-server/vps-strat
receiving incremental file list
rsync: link_stat "/home/user/mybackup/dbonly/2021-11-13" failed: No such file or directory (2)

I have changed the script to match new naming, and no problem so far.

Bottom line, this is probably me being careless again, but I'm glad its not something sinister.