Mouse lag/stutter when remote desktop client (mstsc) full-screen and multiple monitors

On my Windows 10 Enterprise computer, I run Hyper-V. When I connect to a Windows 10 guest vm, I get mouse lag. More accurately it would be a stutter/dropped packet like experience - periodically stopping completely then 'jumping' to the correct spot.

I have two 4K monitors. On one monitor, I maximize the Remote Desktop (RDP) session. The other monitor is displaying my host desktop. When the RDP session to the Windows 10 guest VM has focus, my mouse pointer will stutter when moving over my host desktop. The stuttering continues until I give my host desktop focus. I've tried to capture this in the gif. This is frustrating as it interrupts my workflow.

The left part is monitor 1 displaying the host desktop, the right part is monitor 2 with mstsc maximized. Notice the lag/stutter when passing from the right to the left. It continues for as long as the fullscreen mstsc has focus.

Additional Info:

• Both OS's are latest Windows 10 Enterprise, fully patched
• Issue has been present since day one, clean install of Windows 10 Enterprise
• Both OS's are displaying 3840x2160 (4K). I've tried 1080 with the same issue
• Host computer is fairly powerful; i7-6700K, 16GB DDR4, Radeon RX480 8GB with latest drivers (17.5.2)
• I've tried enabling/disabling RemoteFX on the guest. Also maxed out the video ram on RemoteFX settings
• I've tried Remote Desktop Manager with the same results
• The issue is not present when connecting to Windows 7 or any other operating system
• Oddly enough, the issue is almost reversed when connecting to a Windows 10 Enterprise VM hosted on Azure. Meaning - the stuttering is on the guest when the guest has focus - no lag on my host. This is much worse (VM is too laggy to be useable)
• My network speeds are great. LAN is gigabit. WAN is 100MB. Tested both with iperf.
• I've tried both an internal network connection (IP via static config to hyper-v virtual switch) & an external network connection (IP via DCHP)
• When sharing desktop via Skype, the stuttering becomes consistently worse regardless of focus (RDP or host).
• I've only added mouse trails to help visualize the problem, it doesn't seem to matter
• I've tried changing mouse pointer scheme to 'none' based on a google search result that seemed like a stretch

I've been struggling with this for a while and I'm happy to hear suggestions on things to try or areas to focus.

I was able to resolve this by uninstalling the latest graphics drivers for my video card (AMD) and installing the latest WHQL certified drivers. The drivers were quite old in comparison, but it worked.