Windows 10 Remote Desktop With RemoteFX and Hardware h.264 Using Nvidia NVENC

I have Windows 10 Pro as a server and Windows 10 Pro as a client. Server has a GTX 1070 card with the latest driver and is fully NVENC capable of h.264 / AVC 444 hardware encoding. Steam streaming works using the hardware codec on the server side.

I have RemoteFX enabled on the server, and have enabled the following in the Group Policy under Remote Desktop Services / Remote Desktop Session Host / Remote Session Environment:

• Use hardware graphics adapters for all Remote Desktop Services sessions
• Use advanced RemtoteFX graphics for RemoteApp
• Prioritize H.264/AVC 444 graphics mode for Remote Desktop Connections
• Configure H.264/AVC hardware encoding for Remote Desktop Connections
• Configure compression for RemoteFX data
• Configure image quality for RemoteFX Adaptive Graphics
• Enable RemoteFX encoding for RemoteFX clients designed for Windows Server 2008 R2 SP1
• Enable Remote Desktop Protocol 8.0
• Configure image quality for RemoteFX Adaptive Graphics

When I connect to the server, in the Event Viewer and go to RemoteDesktopServicesd-RdpCoreTS, I don't see any events with EventID 162 or 170, and the documentation says these should appear when hardware encoding is used.

What am I doing wrong? Why am I not getting hardware h.264 encoding? Is there another setting that I'm missing that I need to enable?

Finally managed to get it to work. There were several parts to the solution.

1. Hardware encoding is ONLY ever used with AVC444 encoding. If your client isn't explicitly telling the server it is AVC444 capable (e.g. it if it can only handle AVC420), RDP server will NEVER use hardware encoding. I think this is because MS uses a special codec to generate a stream that is AVC420 decoder correct but applies special encoding that makes fonts not look blocky and blurry.

   Therefore: Make sure your client supports AVC444, and if necessary, don't enable any fallback options.

   If you are using FreeRDP client make sure you start it with /gfx-h264:AVC444, you have set at least /bpp:24, and NO /rfx or /rfx-mode:video. You will also need to make sure you built FreeRDP with ffmpeg h.264 support rather than OpenH.264 as the latter has broken support for AVC444 decoding and will result in artifacts that render RDP unusable.

2. Nvidia driver version matters. I was originally trying with Nvidia's latest "Game Ready" driver, and I couldn't get it to work. In the end, I got it working with the "Studio" driver which is a little older.

3. Hardware support for AVC444 encoding is NECESSARY. That means Maxwell GM107 or better GPU (GeForce 9xx and later only), as older versions don't have YUV444 encoding support. I was initially trying with a GT630 which is based on a Kepler GK208 (GT 630) GPU, which only supports AVC420 and not AVC444.

So to summarize, to make this work you will need:

1. Hardware that supports YUV444/AVC444 encoding (GM107 / 9xx series or later)

2. Nvidia Studio driver (as upposed to the Game Ready driver, at least at the time of writing this)

3. Client that supports AVC444 decoding and reports itself as such.

4. Enable the option to prioritize/prefer AVC444 on the server side.

5. Keep the resolution to <= 2560x1440

Update: this works much better and more stably if instead of Windows 10 you use Windows Server 2016 for your remote gaming host.