ntpdate and ntpd failing to sync clock on Linux

I have a strange problem with one of my servers. ntpd and ntpdate fail to work, but debugging shows no errors at all. At first I thought maybe a local or network firewall was blocking UDP port 123, but that is not the case- this server can talk UDP port 123 (the ntp protocol) to the Internet and get answers.

Let me demonstrate the issue.

date -s "30 DEC 2012 02:30:00" - works, so I can successfully set the clock without error.

ntpq -pn pool.ntp.org - works, I do get detailed time data from the timeserver, and proves UDP packets are functioning.

ntpdate -d pool.ntp.org - debug mode works, shows a ton of debug data and shows the current time offset: 30 Dec 02:38:56 ntpdate[19267]: step time server 208.97.140.69 offset 228.234554 sec

All looks normal, until: ntpdate pool.ntp.org - after a 4.7 second pause, it returns: 30 Dec 02:41:29 ntpdate[19274]: no server suitable for synchronization found

Similar problem running ntpd, it does not update the clock.

After ntpd is started, ntpq -pn results in all refid's forever stuck on .INIT. which means they cannot sync.

/var/lib/ntp/drift is the driftfile setting in ntp.conf, which is chmod 644 and owned by ntp:ntp, same as all my other systems.

I tried a dozen other ntp time servers, disabled iptables firewall, and confirmed the datacenter is not filtering udp traffic. Any ideas what is stopping ntpd and ntpdate from syncing my clock?

This is CentOS 6.3 x64 on a dedicated server with Intel CPU.

ntpdate (and ntpd) will refuse to (easily) set the time if the offset is too high. Both applications will try to slowly adjust your time, so as to not to confuse your system or any applications that may not handle big time jumps very well.

Try ntpdate -b instead. It will set the time no matter how unreasonable it may seem.

You may also need to add the -u flag, which will prevent ntpdate from using privileged (<1024) ports. Note that -u is implied by -d! And it sounds like -d is working fine.

If adding -u makes a difference between working and non-working, then you have a firewall on the way that is causing these issues.

And unfortunately it doesn't seem to be possible to make ntpd use an unrestricted port.

Can you provide the following outputs in pastebin.

cat /etc/ntp.conf
cat /etc/sysconfig/ntpd
ntpq -pn
ntpdc -c sysstat
ntpdc -c kerninfo
ntpdc -c loopinfo
ntpdate -d <time-server-IP>
ntptrace

Are you syncing from stratum 1 servers or anything else.

No server suitable for synchronization means what it tells, that the communication between client and server cannot be established.

If we can't find clues from this set of data, tcpdump might be required to see where the packet is being lost.

tcpdump -s0 -i ethX -p udp -w /tmp/ntp.pcap

Stop and start ntpd daemon and wait for the reach to go to 377 and then stop the tcpdump. That should give further clues.