Websocket based website behind a reverse proxy in IIS
So apparently IIS cannot handle the permessage-deflate (see this question) extension for the Sec-WebSocket-Extensions header. The solution is to clear the header as that is the only extension used by FoundryVTT:
<serverVariables>
<set name="HTTP_SEC_WEBSOCKET_EXTENSIONS" value="" />
</serverVariables>
Do not forget to add HTTP_SEC_WEBSOCKET_EXTENSIONS as an allowed server variable for your site.
Web Socket Reverse, Web Socket Reverse 2 and ReverseProxyOutboundRule1 are not required to make FoundryVTT work with IIS.