MSTSC connection quality vs RDS via Connection Broker connection quality

We have a site in the USA which has a fixed VPN into our Datacentre in the UK via a 100Mb Leaseline.

Our Datacentre Environment is a 2012 R2 Session Collection of RDS Servers, using Microsoft Connection Broker, RDC Gateway, and a Session Collection Web Portal to present a user with an RDP app to then connect into one of the session collection servers (the 'farm')

The Connection Brokers, Gateway Servers, and Farm Servers are all in the same physical location/same LAN in the UK datacentre

All clients are running the latest RDP version 8.1 RemoteFX enabled and in use

The latency from this location to the UK datacentre is 140 ms

When users make use of the Session Collection Portal to connect to the farm they get always get 2 bars, and poor connection quality in the RDP bar. Result is clunky sluggish performance.

If they bypass the web portal, and connect direct to one of the 'farm' servers via MSTSC (remember permanent VPN running between US and UK), their connection quality is Good to Excellent and the resulting experience is perfect.

What are the Session Collection Portal/Connection Broker/Gateway Servers doing to the RDP that could cause this drastic difference as opposed to straight forward MSTSC?

Just for clarification, there are no LAN networking issues in this US location, imagine a single most up to date client, same LAN location, 100 Mb of bandwidth, latency 140ms etc basically nothing changing in any way except using a MSTSC to go direct to one of the servers in the farm, or going via Portal/Connection Brokers/Gateways using the RDP over HTTPS.

Make sure that your RDP gateway is reachable via UDP from the clients. If there's a fallback to TCP and you've high latency and/or some loss, you may get degraded performance and/or connection quality.

Here's some more information about both transports and ports.