Do I need to run a NTP server in every VM?

Couldn't the guests somehow inherit the system time of the host ?

It seems kind of pointless to run the same daemon to get the same results on the same machine multiple times but I didn't find anything related to time when reading KVM or Xen articles. My understanding is that the guest gets the host time on boot but that it might then drift apart. Is that correct ?

This is correct. It should be noted that time not only "can" drift away, but will drift away due to the fact that intervals between timer interrupts (which timekeeping on OS is often based on) are stretched and compressed as the hypervisor would see fit.

A commonly taken workaround across most virtualization platforms (Hyper-V integration services, VMWare tools) is running a daemon on the guest which periodically synchronizes clocks with the VM host. As noted by Hauke in a comment to your question, KVM additionally provides a paravirtualized clock which would need the respective driver loaded on the guest OS to work.

Timekeeping in VMWare Virtual Machines (vmware.com)
KVM guests clock synchronisation (s19n.net)

In a perfect world, your VM guests would keep perfect time, or at least as perfect as the host provides. Unfortunately we don't live in a perfect world.

Based on my experience with virtually every hypervisor known to man, I always run an NTP client in virtual machines, without exception. My usual setup is ntpd with the -g option, or ntpdate starting right before it for old systems, to step the clock (which may be far out of sync at system boot).

KVM has nearly the perfect setup, with its paravirtualized realtime clock; guests with the appropriate driver (all recent Linux, at least) will keep time as well as the host. But still things go wrong here: For instance, the host may not be running NTP, the host may have an incorrect time zone set, the host's clock may just be plain wrong, etc.

VMware and Hyper-V fall in the middle. Each has a tool meant to be run on the guest which syncs the clock with the host periodically, but again, this is vulnerable to any existing problems with the host clock.

Guests on my test Hyper-V server also exhibited some strange behavior: even with integration services, the guest clock would drift faster than 500 ppm, preventing ntpd from working (it considers the clock insane if it drifts faster than this). I had to switch these guests to chrony, which allows this value to be adjusted.

Xen is the worst in this respect; it has absolutely no synchronization and running NTP in the guests is pretty much required. (I am told that very recent versions of Xen have some sort of synchronization but haven't personally worked with it yet.)

Things just get worse if the host hypervisor isn't under your control, such as a public cloud. You are at the mercy of the provider with respect to the host clock, and if they aren't diligent in keeping it synchronized, you lose.

With all that, running NTP clients in your virtual machines is pretty much required if you need even a semi-accurate clock. NB: If you run Windows virtual machines, get a third party NTP client that adjusts the clock continuously; the poor excuse for a client that comes with Windows only adjusts the clock once a week, which is utterly ridiculous.

I would recommend to use NTP because it is well known and has been around a long time. Adjusting the clock is not trivial. NTP has solved this issue.

The official line of VMware is to use one mechanism, with NTP preferred because it is more fine-grained and take smaller steps to adjust the time. The internal VMware solution takes bigger steps. When you run both they could fight each other. With the internal VMware solution taking a big step and then NTP adjusting it and taking it back a bit.

In practice however we run both at the same time and I haven't seen a problem yet.

$ ntpq   
ntpq> peers
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
something.org  172.2.1.5          2 u   57   64  377    1.597   -2.409   5.952


$  vmware-toolbox-cmd  timesync status
Enabled


$ vmware-toolbox-cmd help timesync
timesync: functions for controlling time synchronization on the guest OS
Usage: vmware-toolbox-cmd timesync <subcommand>

Subcommands:
  enable: enable time synchronization
  disable: disable time synchronization
  status: print the time synchronization status