GlusterFS SSL not working
Solution 1:
After spending a few hours trying to figure out what is wrong with the certificates, I found that, in my case, the problem was a stupid mistake: I haven't restarted glusterd on one of the two servers that were hosting the bricks. Both servers were therefore configured exactly the same, but one of them was unaware of the new SSL configuration.
If you encounter the same issue, the first thing is to run service glusterd restart on all the nodes. If it doesn't work, the other answer, as well a bug report may help.
Solution 2:
I have fixed the problem this way (only for I/O path SSL support) ... the cipher list shown in the GlusterFS tutorial did not work it turns out and I have now set the following:
gluster volume set mailrep-volume ssl.cipher-list 'HIGH:!SSLv2:!SSLv3:!TLSv1:!TLSv1.1:TLSv1.2:!3DES:!RC4:!aNULL:!ADH'
Also I did not concate the .pem files from the servers but instead created my own self-signed CA as shown in the more complex example above. The .ca file created this way must be the same on all peers and when it is used to sign the glusterfs.pem files no concat certs are needed.
This is my complete volume config:
Volume Name: mailrep-volume
Type: Replicate
Volume ID: 907250fb-d83e-4fe7-a63b-636d952095ff
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eris:/mnt/glusterfs/vmail
Brick2: orcus:/mnt/glusterfs/vmail
Options Reconfigured:
ssl.cipher-list: HIGH:!SSLv2:!SSLv3:!TLSv1:!TLSv1.1:TLSv1.2:!3DES:!RC4:!aNULL:!ADH
nfs.disable: on
transport.address-family: inet
client.ssl: on
server.ssl: on
auth.ssl-allow: eris,orcus
To verify I can do ... (1) and see ... (2)
(1) cat /var/log/glusterfs/glustershd.log | grep SSL
(2) 0-mailrep-volume-client-1: SSL support on the I/O path is ENABLED
(2) 1-mailrep-volume-client-1: SSL support on the I/O path is ENABLED
The error from above (SSL3_GET_RECORD:wrong version number) is fixed now ONLY when using SSL on the I/O path. The problem still exists when I enable SSL on the management path but I don't know how to alter the cipher-lists for that to apply the same fix.
A little side note: The GlusterFS SSL tutorial seems to be horribly outdated. It is also stated that /etc/ssl/glusterfs.* are hardcoded file locations, but at least for SSL on the I/O path you can easily apply custom locations this way (tested and worked):
gluster volume set mailrep-volume ssl.private-key /path/to/your/certkey
gluster volume set mailrep-volume ssl.own-cert /path/to/your/certfile
gluster volume set mailrep-volume ssl.ca-list /path/to/your/cafile
I found those options in the Gluster source.
PS: I have also upgraded to GlusterFS 3.12.0 today (after the fix!)