tunneling secure websocket connections with apache

I have an Apache running that is only accessible via HTTPS. I want to serve websockets from an additional server application which runs on the same machine, but since it is not possible for clients to connect on another port than 443 to our server, those websocket connections need to be proxied through the Apache.

Now, I've installed mod_proxy and configured it as follows:

SSLProxyEngine on
ProxyPass /ws https://127.0.0.1:9001

This does not work however. I can connect to https://server/ws in my browser now, but the apache seems to swallow part of the websockets headers, so that real websocket connections do not work.

How can I accomplish tunneling my websocket connections through the Apache server?

Solution 1:

I've got it working.

Scenario

-------------       ----------------       ----------
| Browser   |<----->| Apache httpd |<----->| Tomcat |
|           |  SSL  |    2.4.9     |  SSL  | 7.0.52 |
-------------       ----------------       ----------

Browser WebSocket through Apache httpd, reverse proxying to the web app in Tomcat. All SSL front-to-back.

Here's the configuration for each piece:

Browser Client

Note the trailing "/" in the url: wss://host/app/ws/. It was necessary to match the correct wss ProxyPass directive (shown further down in the Apache config section) and preventing a 301 redirect to https://host/app/ws. That is, it was redirecting using the https scheme and not the wss scheme for the back-end.

Test Page

<!doctype html>
<body>

<script type="text/javascript">
    var connection = new WebSocket("wss://host/app/ws/");

    connection.onopen = function () {
        console.log("connected");
    };

    connection.onclose = function () {
        console.log("onclose");
    };

    connection.onerror = function (error) {
        console.log(error);
    };
</script>

</body>
</html>

Apache httpd

I am using Apache httpd 2.4.9, which out of the box provides mod_proxy_wstunnel. However, the mod_proxy_wstunnel.so provided does not support SSL when using wss:// scheme. It ends up trying to connect to the back-end (Tomcat) in plaintext, which fails the SSL handshake. See bug here. So, you have to patch mod_proxy_wstunnel.c yourself by following the suggested correction in the bug report. It's an easy 3 line change.

Suggested correction,
314a315
>     int is_ssl = 0;
320a322
>         is_ssl = 1;
344c346
<     backend->is_ssl = 0;
---
>     backend->is_ssl = is_ssl;

Then rebuild the module and replace in your new mod_proxy_wstunnel.so with the old one.

Building Apache httpd

Here's the (2.4.9) command I used to build in the modules I wanted. You might not need them all.

./configure --prefix=/usr/local/apache --with-included-apr --enable-alias=shared
--enable-authz_host=shared --enable-authz_user=shared 
--enable-deflate=shared --enable-negotiation=shared 
--enable-proxy=shared --enable-ssl=shared --enable-reqtimeout=shared
--enable-status=shared --enable-auth_basic=shared
--enable-dir=shared --enable-authn_file=shared
--enable-autoindex=shared --enable-env=shared --enable-php5=shared
--enable-authz_default=shared --enable-cgi=shared
--enable-setenvif=shared --enable-authz_groupfile=shared
--enable-mime=shared --enable-proxy_http=shared
--enable-proxy_wstunnel=shared

Note the very last switch: --enable-proxy_wstunnel=shared At first, I was incorrectly using --enable-proxy-wstunnel=shared, which seemed to build fine, but ultimately wouldn't work when I used the resultant .so file. See the difference? You want to make sure to use an underscore in "proxy_wstunnel" rather than a dash.

Apache httpd config

httpd.conf

...
LoadModule proxy_module modules/mod_proxy.so
...
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
...
LoadModule ssl_module modules/mod_ssl.so
...
Include conf/extra/httpd-ssl.conf
...
LogLevel debug
ProxyRequests off

# Note, this is the preferred ProxyPass configuration, and *should* be equivalent
# to the same inline version below, but it does NOT WORK!
#<Location /app/ws/>
#        ProxyPass wss://localhost:8443/app/ws
#        ProxyPassReverse wss://localhost:8443/app/ws
#</Location>
#<Location /app/>
#        ProxyPass https://localhost:8443/app/
#        ProxyPassReverse https://localhost:8443/app/
#</Location>

# NOTE: Pay strict attention to the slashes "/" or lack thereof!
# WebSocket url endpoint
ProxyPass /app/ws/ wss://localhost:8443/app/ws
ProxyPassReverse /app/ws/ wss://localhost:8443/app/ws

# Everything else
ProxyPass /app/ https://localhost:8443/app/
ProxyPassReverse /app/ https://localhost:8443/app/

If you didn't see my note in the above config, here it is again: Pay strict attention to the slashes "/" or lack thereof!

Also, if you are seeing debug log statements in your apache log that says a wss connection was made then closed, it is possible that you have mod_reqtimeout enabled as I did, so make sure it not loaded:

#LoadModule reqtimeout_module modules/mod_reqtimeout.so

Tomcat

Assuming your HTTP connector is setup correct, there's not much to configure in tomcat. Though to aid in debugging, I found it useful to create a $CATALINA_HOME/bin/setenv.sh that looked like this:

setenv.sh

CATALINA_OPTS=$CATALINA_OPTS" -Djavax.net.debug=all -Djavax.net.debug=ssl:handshake:verbose"

This allowed me to see if the mod_proxy_wstunnel.so that I modified was working or not for wss://. When it wasn't working, my catalina.out log file would show:

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
http-nio-8443-exec-1, SEND TLSv1 ALERT:  fatal, description = internal_error
http-nio-8443-exec-1, WRITE: TLSv1 Alert, length = 2
http-nio-8443-exec-1, called closeOutbound()
http-nio-8443-exec-1, closeOutboundInternal()

Final Thoughts

Though I am using Apache httpd 2.4.9, I've seen where backports of mod_proxy_wstunnel can be applied to versions 2.2.x. Hopefully my notes above can be applied to those older versions.

Solution 2:

If you don't want Apache to terminate the SSL connection (and forward unencrypted WebSocket traffic), but have the SSL terminated on the final target WebSocket server and exlusively want to use WSS on the WebSocket traffic coming into Apache, then mod_proxy_connect may be able to just connect through the raw traffic. Not sure. I'd be also interested if that works.

If above does not hold, here is more information:

https://serverfault.com/questions/290121/configuring-apache2-to-proxy-websocket
https://issues.apache.org/bugzilla/show_bug.cgi?id=47485
http://blog.alex.org.uk/2012/02/16/using-apache-websocket-to-proxy-tcp-connection/

In any case, using Apache will severly limit the scalability regarding number of concurrently served WebSocket connections, since every WS connection will consume 1 process/thread on Apache.