Sudden system time jumps on VM

for the second day in a row the system time on a VM I'm in charge of jumped forward by exactly 90026 seconds. In the afternoon or evening, not connectable to any cron maintenance job which typically run 3-5 am.

I don't know where to begin looking. /var/log/messages just says "systemd: Time has been changed", and ntpd of course quits because it's too scared to correct such a large deviation.

Any idea how to find out which process might be at fault?

Solution 1:

Ninety thousand seconds is 25 hours, an enormous offset that sets the time to the wrong day.

The clock actually being stepped while the system is running, but chrony panics, suggests it was not done by NTP. Confirm your NTP servers are a correct reference by reviewing the output of chronyc tracking

Most likely remaining possibility is hypervisor syncing host time to the guest. Which happens at guest boot, resume, and possibly other occasions depending on platform. VM host time could be horribly wrong due to never having NTP set up correctly. Your compute administrators will need to point those hosts at NTP servers like every other device.

Yes, you can brute force correct it with chrony's maxchange. Beware stepping the time so far can make some applications misbehave.