Ubuntu 20.04 time sync problems and possibly incorrect status information
I have been having some problems with crashes on my KVM host (Lubuntu 20.04), and when troubleshooting, I noticed some time-related errors.
Upon further investigation, to my horror, I saw that time was not being synced. I am sure it was set up before, I have no clue how it became un-setup.
admin@virtland:~$ sudo timedatectl
[sudo] password for admin:
Local time: Fri 2020-07-10 09:14:14 EDT
Universal time: Fri 2020-07-10 13:14:14 UTC
RTC time: Fri 2020-07-10 13:14:14
Time zone: America/New_York (EDT, -0400)
System clock synchronized: no
NTP service: n/a
RTC in local TZ: no
admin@virtland:~$
I found this thread and tried the top answer, but no no avail. https://askubuntu.com/questions/929805/timedatectl-ntp-sync-cannot-set-to-yes
admin@virtland:~$ sudo systemctl stop ntp
admin@virtland:~$ sudo ntpd -gq
10 Jul 09:17:57 ntpd[34358]: ntpd [email protected] (1): Starting
10 Jul 09:17:57 ntpd[34358]: Command line: ntpd -gq
10 Jul 09:17:57 ntpd[34358]: proto: precision = 0.070 usec (-24)
10 Jul 09:17:57 ntpd[34358]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): good hash signature
10 Jul 09:17:57 ntpd[34358]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): loaded, expire=2020-12-28T00:00:00Z last=2017-01-01T00:00:00Z ofs=37
10 Jul 09:17:57 ntpd[34358]: Listen and drop on 0 v6wildcard [::]:123
10 Jul 09:17:57 ntpd[34358]: Listen and drop on 1 v4wildcard 0.0.0.0:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 2 lo 127.0.0.1:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 3 enp6s0 10.0.0.18:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 4 lo [::1]:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 5 enp6s0 [fe80::7285:c2ff:fe65:9f19%3]:123
10 Jul 09:17:57 ntpd[34358]: Listening on routing socket on fd #22 for interface updates
10 Jul 09:17:58 ntpd[34358]: Soliciting pool server 209.50.63.74
10 Jul 09:17:59 ntpd[34358]: Soliciting pool server 4.53.160.75
10 Jul 09:18:00 ntpd[34358]: Soliciting pool server 69.89.207.199
10 Jul 09:18:00 ntpd[34358]: Soliciting pool server 72.30.35.88
10 Jul 09:18:01 ntpd[34358]: Soliciting pool server 173.0.48.220
10 Jul 09:18:01 ntpd[34358]: Soliciting pool server 162.159.200.1
10 Jul 09:18:01 ntpd[34358]: Soliciting pool server 108.61.73.243
10 Jul 09:18:02 ntpd[34358]: Soliciting pool server 208.79.89.249
10 Jul 09:18:02 ntpd[34358]: Soliciting pool server 208.67.75.242
10 Jul 09:18:02 ntpd[34358]: Soliciting pool server 91.189.94.4
10 Jul 09:18:03 ntpd[34358]: Soliciting pool server 91.189.89.198
10 Jul 09:18:03 ntpd[34358]: Soliciting pool server 67.217.112.181
10 Jul 09:18:04 ntpd[34358]: Soliciting pool server 91.189.89.199
10 Jul 09:18:04 ntpd[34358]: Soliciting pool server 64.225.34.103
10 Jul 09:18:05 ntpd[34358]: Soliciting pool server 91.189.91.157
10 Jul 09:18:06 ntpd[34358]: Soliciting pool server 2001:67c:1560:8003::c8
10 Jul 09:18:06 ntpd[34358]: ntpd: time slew +0.001834 s
ntpd: time slew +0.001834s
admin@virtland:~$ sudo service ntp start
admin@virtland:~$ sudo timedatectl
Local time: Fri 2020-07-10 09:18:21 EDT
Universal time: Fri 2020-07-10 13:18:21 UTC
RTC time: Fri 2020-07-10 13:18:21
Time zone: America/New_York (EDT, -0400)
System clock synchronized: no
NTP service: n/a
RTC in local TZ: no
admin@virtland:~$
I thought maybe I needed to use some more up-to-date instructions, so I tried this: https://linuxconfig.org/how-to-sync-time-on-ubuntu-20-04-focal-fossa-linux
admin@virtland:~$ timedatectl set-ntp off
Failed to set ntp: NTP not supported
admin@virtland:~$ timedatectl set-ntp on
Failed to set ntp: NTP not supported
Then I tried this, from a different thread:
admin@virtland:~$ sudo systemctl status systemd-timesyncd.service
[sudo] password for admin:
● systemd-timesyncd.service
Loaded: masked (Reason: Unit systemd-timesyncd.service is masked.)
Active: inactive (dead)
I have never touched timesyncd.conf, but it is entirely commented out anyway:
cat /etc/systemd/timesyncd.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.
[Time]
#NTP=
#FallbackNTP=ntp.ubuntu.com
#RootDistanceMaxSec=5
#PollIntervalMinSec=32
#PollIntervalMaxSec=2048
I checked timedatectl again, and now it is on, but still not using NTP. I understand that NTP is more precise, and that can be important in some situations. Not sure if virtualization with pci passthrough needs extremely precise time or not.
From other stuff I was reading, I thought maybe NTP was conflicting with timesyncd. So remove ntp for the time being:
sudo systemctl stop ntp
sudo apt-pruge ntp
But after purging ntp, NTP showed as active!
admin@virtland:~$ timedatectl
Local time: Fri 2020-07-10 09:34:52 EDT
Universal time: Fri 2020-07-10 13:34:52 UTC
RTC time: Fri 2020-07-10 13:34:52
Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
Am I going crazy? Is NTP still here somehow? Nope.
admin@virtland:~$ sudo systemctl start ntp
Failed to start ntp.service: Unit ntp.service not found.
Apologies for not asking a more focused question, but what the heck is going on here?
I am well and truly lost. Also, I will edit this post later and make a not as to whether removing NTP (and thus activating it?!) fixed the stability problems that led me down this rabbit hole.
Edit: The next thing I did was disable ntp on timesyncd and (re)install NTP as described here. https://www.digitalocean.com/community/tutorials/how-to-set-up-time-synchronization-on-ubuntu-18-04
That resulted in:
admin@virtland:~$ ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
0.us.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
1.us.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
2.us.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
3.us.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
0.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
1.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
2.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
3.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
ntp.ubuntu.com .POOL. 16 p - 64 0 0.000 0.000 0.000
admin@virtland:~$ timedatectl
Local time: Fri 2020-07-10 10:35:39 EDT
Universal time: Fri 2020-07-10 14:35:39 UTC
RTC time: Fri 2020-07-10 14:35:40
Time zone: America/New_York (EDT, -0400)
System clock synchronized: no
NTP service: n/a
RTC in local TZ: no
admin@virtland:~$ systemctl status systemd-timesyncd
● systemd-timesyncd.service
Loaded: masked (Reason: Unit systemd-timesyncd.service is masked.)
Active: inactive (dead)
admin@virtland:~$ nano /etc/ntp.conf
admin@virtland:~$ systemctl status systemd-timesyncd
● systemd-timesyncd.service
Loaded: masked (Reason: Unit systemd-timesyncd.service is masked.)
Active: inactive (dead)
admin@virtland:~$ ntpstat
unsynchronised
polling server every 8 s
I reversed those changes as recommended my Michael Hampton: Does this mean it's working?
boss@virtland:~$ sudo systemctl stop ntp
Failed to stop ntp.service: Unit ntp.service not loaded.
boss@virtland:~$ sudo timedatectl set-ntp yes
boss@virtland:~$ sudo timedatectl set-ntp on
boss@virtland:~$ ntpq -p
bash: /usr/bin/ntpq: No such file or directory
boss@virtland:~$ systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; >
Active: active (running) since Fri 2020-07-10 10:49:18 EDT; 50s ago
Docs: man:systemd-timesyncd.service(8)
Main PID: 108365 (systemd-timesyn)
Status: "Initial synchronization to time server 91.189.94.4:123 (ntp.ubu>
Tasks: 2 (limit: 154317)
Memory: 1.8M
CGroup: /system.slice/systemd-timesyncd.service
└─108365 /lib/systemd/systemd-timesyncd
Jul 10 10:49:17 virtland systemd[1]: Starting Network Time Synchronization...
Jul 10 10:49:18 virtland systemd[1]: Started Network Time Synchronization.
Jul 10 10:49:18 virtland systemd-timesyncd[108365]: Initial synchronization t>
lines 1-14/14 (END)
timedatectl
Local time: Fri 2020-07-10 10:52:56 EDT
Universal time: Fri 2020-07-10 14:52:56 UTC
RTC time: Fri 2020-07-10 14:52:56
Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
So I guess it is working. Since the crashes that took me down this path are still happening, I guess the time wasn't the issue.
Solution 1:
The timedatectl command reports that NTP is in use when either chronyd or systemd-timesyncd is enabled and running. Both of these are lightweight NTP clients suitable for most servers and workstations. It does not recognize the ntpd service, which is designed to be both an NTP client and server, and now that we have dedicated NTP client software, is no longer favored for systems that don't need to serve NTP onward to other systems or use specialized hardware such as GPS to obtain the time.