Restrict access to RD Gateway based on IP

I'm trying to restrict who can access our RD Gateway based on both their group membership and IP address (so people in group A can only access the system from IP address X). The Network Policy Server installed by the RD Gateway seems to imply that this can be done, there is a setting to restrict access based on Client IP Address, however this does not seem to work correctly.

If I add an IP address restriction, the users cannot connect even if they have the right IP, removing that restriction means they can then connect. Looking at the audit log, it appears that the IP address isn't present there.

Does anyone know how to make this work?

Unfortunately, this is not possible.

Seen from the NPS, the radius client is the RDS gateway server (it's the client that actually requests auth and/or accounting from the NPS/Radius server). The client machine connecting to the RDS gateway is therefore not visible to the radius server.

This is much the same on other devices (an example would be an access point) where the client is the device that is forwarding the auth/acc request to the NPS/radius server, rather than the device that is actually trying to connect.

You can limit the access on network layer - but at this point, you cannot distinguish between different user groups.