What is a realistic average time difference between servers in the same LAN?

NTP usually hovers in the <10-20ms range on a "reasonable" network. If you're using Windows built-in NTP client however, that's a different story. The Windows built-in client is only meant to keep the time synchronized enough for Kerberos, 5 minutes. They had no intention of keeping very accurate time; so 100ms is doing pretty good.

If you need more accurate time than 100ms, install an actual NTP client, like the Meinberg binaries of ISC's Reference NTPd.

If you need ultra accurate time, you should be looking at Precision Time Protocol (PTP). Be warned that PTP is much more complicated than NTP, so unless you need ultra accurate timestamping or something similar, NTP is highly advised.

NTP ought to be able to do a lot better than 100ms

Virtualized machines are notorious for having odd time sync issues.

You don't mention your virtualization environment, but check and make sure all the hypervisors are NTP sync'd as well. (Generally it is best for a guest VM machine to take its clock from its host hypervisor.)

See this 'Known Issue' in the NTP documentation -- it gives some more detail.

http://support.ntp.org/bin/view/Support/KnownOsIssues#Section_9.2.2.

See this paper from VMware for excruciating detail about problems with timekeeping on Windows hardware in general and additional problems with timekeeping in virtual machines:

http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf