Causes of RTP jitter at the server

Finally figured this out! TLDR: disable power management on the host.

Despite the low CPU usage, we still figured this was something to do with CPU load. So, we were experimenting with loading down the CPU, expecting this problem with the dead spots in the calls to get worse. Instead, it went away completely. So, after looking at CPU usage stats in vCenter many many times, I finally looked into the other line on that graph.

This is probably not news to many, but I found out that CPU ready time is the amount of time that a VM is ready to use the CPU, but the physical resources can't be allocated by the host. Most sources I found say that anything less than 5% isn't a problem, but it certainly seemed to be having an impact on our voice streams. We were seeing the cutouts every minute, and the graph also showed a spike in ready time every minute.

So I got to wondering why this would go away during high CPU load and figured it must be some kind of power management. When the host sees the increased usage it makes CPU resources consistently available to the VM. So I disabled power management in the BIOS of the host, et voila:

^{_{The slight increase in ready time near the end of the graph corresponds to a half-dozen other VMs migrating back to this host.}}

Call traces now show negligible amounts of jitter, and the cutouts have disappeared from calls. Further research showed this is a somewhat common issue with workloads that are both latency-sensitive and CPU non-intensive. The power management sees the very low CPU usage and assumes it can throttle the processor, even though it should not!