Erratic mouse movement in 3D games over RDP with RemoteFX

It seems that the default mouse driver when connecting with RDP does not work well with certain applications, such as 3D games.

My setup:

• Windows 2012 R2 with Hyper-V and strong graphics card
• VM with Windows 8.1 Enterprise, RemoteFX and vGPU configured
• Connecting with RDP over fast link.

3D video performance is great, thanks to RemoteFX/vGPU. It definitely allows games to be very playable over RDP (if it weren't for the mouse).

The problem is that the mouse reacts quite crazily when in game - but completely normal outside of the 3D environment. (E.g. in the game menus the mouse is just fine).

Closest I could tell, from both experimentation and much searching online (many other people had the same problem, but no solution found) - it seems the mouse driver transmits its relative location, instead of movement.
This "theory" correlates well with some explanations of how the mouse pointer jumps in and out of the RDP window (e.g. move it out the window on the left, back in on the right - and it behaves perfectly, i.e. location-based mouse and not motion-based), and also with the in-game behavior - e.g. if I move the mouse a bunch to the left, the view jumps crazily to the left; if I then move it a bit to the right, but still not all the way back to center (i.e. the mouse is still to the left of center, but moving to the right) it continues to jump left. (I.e. it seems it is still reporting it's location "left of center" instead of "moving to the right").

I am hoping to find ANY kind of solution to this. Ideas anyone?

Wow, after a ton of research and failed attempts, I actually solved this!
Well, mostly solved it - it is functional, but not without drawbacks.

Thanks to @JourneymanGeek's answer, and in particular the post he linked to, I was able to follow the lead, and eventually discovered that amongst the many interfaces implemented by the Remote Desktop Client ActiveX, one of them supports a RelativeMouseMode property! That sounds like exactly what I need, it will force the RDP to support relative mouse movements!

However, I did note on this MSDN Forums post that "RelativeMouseMode is not supported in RDP RDSH/RDVH scenarios and should not be used", but I figured to heck with it, it's not a real production environment, and I was fine using a feature that is not supported. It was also poorly documented, but it seemed that I had what I needed - this question on StackOverflow also gave me hope that it was doable.

So, I set to work implementing a simple WinForms app to host the ActiveX control, with the "...Unsafe" interface properties set.

Except that as it turns out, by "unsupported", this time Microsoft meant "it does not work".
Well, if it did, this would be an SO question, sorry to lead you on (but I think its good to have it documented somewhere...)

However, not all was for naught, during all that digging into the RDP protocol, I looked at the RemoteFX USB Redirection - and that looked like a dead-end, since basic input devices (such as mouse, keyboard, printer) are explicitly blocked from the USB Redirection mechanism:

By default, devices in the aforementioned categories are accessible in the remote session by using high-level device redirection methods. These methods of redirection enable optimal performance and backward compatibility of the device in the majority of user scenarios. Consequently, these devices are not offered via RemoteFX USB redirection.

Well, the final answer, as it turns out, is that there is an override mechanism to this block.

Step 1: Enable RemoteFX USB Redirection
There is a nice elaboration here: https://www.itprotoday.com/server-virtualization/q-how-do-i-enable-remotefx-usb-redirection . Basically, on the client machine, you use either Local Policy or Group Policy to set RemoteFX USB Device Redirection to Enable, and allow users (or just admins) the rights. Then gpupdate /force and reboot.

Step 2: Enable the redirection override for the mouse.
As this MS KB article explains, you can set a registry key to enable a specific device (or class of device) for USB Redirection.

Again on the client, under

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services\Client\UsbSelectDeviceByInterfaces

you can add a string value with the device identifier.
For example, under this key I added the following string value:

"GUID_DEVINTERFACE_MOUSE"="{378DE44C-56EF-11D1-BC8C-00A0C91405DD}"

Depending on your mouse, you might need a different identifier, so YMMV. But start with that.

Step 3: Set the RDP client to redirect the mouse.
As the article from step 1 stated:

Once it's enabled, access the Local Resources tab, click More under Local devices and resources, and you'll see a new Other supports RemoteFX USB devices setting. You can use this setting to map through many types of USB hardware that you couldn't with vanilla RDP USB redirection, as shown below.

Under that you should see an entry for your mouse; it might be a bit ambiguous such as:

Select your mouse (or other appropriate entry).

That's it! You now have a hardware mouse attached "directly" to your VM, which allows the 3D games to pull out the relative mouse movements directly from the mouse driver (-ish..)

However, as stated, there are some drawbacks.

1. The mouse is redirected to the RemoteFX'd VM - in other words, the client machine no longer has a mouse, at least while the Remote Desktop session is open.
   A possible solution would be to attach a 2nd mouse, and redirect only one of them. (I have yet to try this, but it should work just fine).

2. For some reason, when redirected the mouse does not directly show a cursor in the remote session. While it still works, it does make it a bit confusing to aim what your mouse is pointing at.
   A simple solution is to turn on Mouse Trails, with the length turned down all the way.
   This will have the mouse location show up, but it is a bit sluggish.

3. Overall, while this solution does work well, the mouse is still a bit slow and sluggish.
   You can turn up the mouse speed, play with pointer precision (sometimes turning it OFF will actually help with this), and even twiddle with the mouse acceleration registry settings under HKEY_CURRENT_USER\Control Panel\Mouse (Mouse Speed, MouseThreshold1, and MouseThreshold2 respectively).
   This will help mitigate the problem, but not really make it go away.

So overall, this can make even 3D games very playable - for casual gaming, unfortunately not great for twitch gaming. (Though if you need that you can just boot metal straight into that VHD, and have both options....)

Apparently remotefx does tablet style 'absolute' input modes only, which is the source of your issue. I can't find any documentation on changing it.

You may need to consider an alternate remote access mode - steam works well for me (using an nvidia GPU on the 'host' and a dinky little bay trail box on the other end). Nvidia has an option for their cards, but I've not really tried that.

In short, there's no obvious way to fix this with remotefx, unless the game could be set to use the input mode remotefx is using.

Remote Desktop transmits absolute mouse movements, as a way to reduce the number of mouse movement packets that need to be sent from client to server. This arrangement works well for most windows applications (email, web browsing etc). Many 2D games are also fine with this. However, FPS games generally need relative mouse movements in order to control mouselook, which Remote Desktop unfortunately does not transmit. The loss of relative mouse movement data is what causes the game to rotate the viewport unexpectedly. Microsoft would need to modify Remote Desktop in order to make this gaming scenario work. However, using the XBOX controller for PC with USB redirection might be an alterative for you.