What can be done to improve time synchronization on networks with sporadic internet access?

I'm looking for advice setting up time servers for a very non-typical network. I support many closed networks that have occasional access to the internet. A network would get access most days for a few hours, but would frequently go 1-3 weeks blacked-out. The computers/servers on this network are mostly *nix-based, but not all the same flavor. The entire network is mobile, so when it connects, it will have very different hops/latency to internet time servers. The servers on the closed network are powered-off frequently (at least daily). Right now, my gut tells me to use NTP (because I hate re-learning all the stuff that someone else already got working pretty well). But I have several issues, and am looking for someone with experience in this type of strange situation. I currently have no solution in place, I'm simply letting the internal clocks drift. This results in errors of ~600s in a majority of networks. I have seen mismatch worse than 10,000s.

  1. Is there something "better" than NTP in this situation? I know NTP likes to have very frequent, consistent access to servers that give nearly identical answers. I won't have that.
  2. How many internal NTP servers should I configure, so that during periods of internet blackout, I have internal time that is consistent within the closed network?
  3. There is no human access. No matter how large the mismatch, the server(s) must attempt to correct itself.
  4. Discrete steps are very bad. No matter how large the mismatch, the correction must be "slewed", not "stepped". I understand that this could take many hours to correct.

Solution 1:

In the old days, setting up a stratum-1 NTP server was very difficult, because stratum-0 sources were very expensive, extremely delicate, and usually radioactive.

Nowadays we have the GPS, which incidentally functions as an extremely accurate radio clock. You can buy a dedicated stratum-1 server containing a GPS receiver as its stratum-0 timesource for quite reasonable prices, or with a little ingenuity you can attach a decent consumer-grade GPS unit to a random server, and set up NTP accordingly to give you your own stratum-1 server. The first of those is better for improved availability.

Do one of those things, and you'll have a single stratum-1 server on your network to which everyone can sync. One is enough; everyone will be sync'ed to it while it's up, and if it goes down, all the clients will probably have got a good idea of their drift rates, so they shouldn't drift too far before it comes back, at which time they'll gently resync to it.

I can't see any reason not to have good time even with intermittent internet access.