Windows Server 2012 - RDP over UDP not working
I have a Windows Server 2012 (not R2) machine, hosted within Hyper-V virtualization with RD Session Host & RD Gateway installed. It is used to run a desktop GIS application.
The performance over WAN is quite poor. I added UDP port to NAT in order to improve the performance, but the UDP connection is still not used.
In the LAN-testing environment (to avoid NAT / firewall misconfigurations), I connect from a Win10 machine. The Connection Info bar shows "excellent quality", but does not mention anything about UDP. In case of the reverse connection (Windows Server => Windows 10), the connection info bar says that UDP is enabled.
I completely turned off Windows Firewall installed on the Windows Server. I double-checked that UDP transport is enabled in RD Gateway. There is no difference if I connect using Gateway (443+3391) or Direct (3389+3389). I restarted whole machine twice and scrolled over all links in first three google result pages.
Do you have any idea what can be wrong?
I suspect a routing error. Run a wireshark to be sure the UDP packet get received. As I seen it's over a WAN, router usually does not allow well the packets.
1.5 Prerequisites/Preconditions
The protocol endpoints require UDP connectivity to be established. The network path between the endpoints should allow the transfer of UDP datagrams in both directions.
The prerequisites for this protocol are identical to those for the UDP protocol.
1.3.2.1
UDP Connection Initialization In this phase, both endpoints are initialized with mutually agreeable parameters for the connection. The terminal client initiates the connection by sending a SYN datagram. The terminal client also determines the mode of operation, RDP-UDP-R or RDP-UDP-L, as described in section 1.3.1. The terminal server responds with a datagram with the SYN flag set, along with an ACK flag, to acknowledge the receipt of the SYN datagram. The terminal client acknowledges the SYN datagram by sending an ACK. The terminal client can append the Coded Packets along with the ACK datagram. This datagram indicates that a connection has been set up and data can be exchanged. All datagrams in this phase – the SYN, SYN+ACK, and ACK – are delivered reliably by using persistent retransmits, irrespective of the mode that the transport is operating in.
Have you checked that UDP is able to pass through the host Hyper-v server?
If possible try using an alternate virtual network adapter type in the virtual switch manager and/or that the physical network adaptor config settings to identify any UDP related settings that maybe off.
I chased this issue around for quite a while and coudn't find the solution elsewhere. Same situation as yours, connecting directly to the RDS Server didn't work with UDP transport. UDP 3391 was listening and not blocked by firewall. Other non-RDS servers did work with UDP transport.
Configured the following GPO, ran gpupdate /target:Computer /force, and voila, it's working.
Set Computer Configuration > Remote Desktop Services > Remote Desktop Session Host > Security > Require use of specific security layer for remote (RDP) connections to Enabled (SSL)