Fix 'TLS Error: TLS handshake failed' on OpenVPN client
I am configuring OpenVPN 2.3.6-1 on my Arch Linux server in order to encrypt SMB traffic over the public Internet. When I test the setup on one of my Linux virtual machine clients, I get the error: TLS Error: TLS handshake failed.
I quickly read (OpenVPN on OpenVZ TLS Error: TLS handshake failed (google suggested solutions not helping)) and tried to switch from the default UDP to TCP, but that only caused the client to repeatedly report that the connection timed out. I also tried disabling the cipher and TLS authentication, but that caused the server to fail with Assertion failed at crypto_openssl.c:523. In both instances, the required changes were made to both the client and server configurations.
I have been following the instructions at (https://wiki.archlinux.org/index.php/OpenVPN) to set up OpenVPN and the instructions at (https://wiki.archlinux.org/index.php/Create_a_Public_Key_Infrastructure_Using_the_easy-rsa_Scripts) to create the keys and certificates. The only deviations I have made from these instructions have been specifying my own computers' names and their corresponding key/certificate file names.
See also my original question about securing SMB traffic over the Internet: (Simple encryption for Samba shares)
Can anybody explain how I can solve this issue?
Details:
Server: Arch Linux (up to date) connected directly to gateway via ethernet cable. No iptables.
Client: Arch Linux (up to date) virtual machine on VirtualBox 4.3.28r100309 Windows 8.1 host, bridged network adapter. No iptables. Windows Firewall disabled.
Gateway: Port forwarding for port 1194 enabled, no firewall restrictions.
Here are the configuration files on the server and client, respectively. I created these according to the instructions on the Arch Wiki.
/etc/openvpn/server.conf (Non-comment lines only):
port 1194
proto udp
dev tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server-name.crt
key /etc/openvpn/server-name.key
dh /etc/openvpn/dh2048.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
tls-auth /etc/openvpn/ta.key 0
comp-lzo
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
verb 3
/etc/openvpn/client.conf (Non-comment lines only):
client
dev tun
proto udp
remote [my public IP here] 1194
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client-name.crt
key /etc/openvpn/client-name.key
remote-cert-tls server
tls-auth /etc/openvpn/ta.key 1
comp-lzo
verb 3
Here are the outputs of running openvpn on the machines with the above configurations. I started the server first, then the client.
The output of openvpn /etc/openvpn/server.conf on the server:
Thu Jul 30 17:02:53 2015 OpenVPN 2.3.6 x86_64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Dec 2 2014
Thu Jul 30 17:02:53 2015 library versions: OpenSSL 1.0.2d 9 Jul 2015, LZO 2.09
Thu Jul 30 17:02:53 2015 NOTE: your local LAN uses the extremely common subnet address 192.168.0.x or 192.168.1.x. Be aware that this might create routing conflicts if you connect to the VPN server from public locations such as internet cafes that use the same subnet.
Thu Jul 30 17:02:53 2015 Diffie-Hellman initialized with 2048 bit key
Thu Jul 30 17:02:53 2015 Control Channel Authentication: using '/etc/openvpn/ta.key' as a OpenVPN static key file
Thu Jul 30 17:02:53 2015 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 17:02:53 2015 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 17:02:53 2015 Socket Buffers: R=[212992->131072] S=[212992->131072]
Thu Jul 30 17:02:53 2015 ROUTE_GATEWAY 192.168.0.1/255.255.255.0 IFACE=enp5s0 HWADDR=##:##:##:##:##:##
Thu Jul 30 17:02:53 2015 TUN/TAP device tun0 opened
Thu Jul 30 17:02:53 2015 TUN/TAP TX queue length set to 100
Thu Jul 30 17:02:53 2015 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Thu Jul 30 17:02:53 2015 /usr/bin/ip link set dev tun0 up mtu 1500
Thu Jul 30 17:02:53 2015 /usr/bin/ip addr add dev tun0 local 10.8.0.1 peer 10.8.0.2
Thu Jul 30 17:02:53 2015 /usr/bin/ip route add 10.8.0.0/24 via 10.8.0.2
Thu Jul 30 17:02:53 2015 GID set to nobody
Thu Jul 30 17:02:53 2015 UID set to nobody
Thu Jul 30 17:02:53 2015 UDPv4 link local (bound): [undef]
Thu Jul 30 17:02:53 2015 UDPv4 link remote: [undef]
Thu Jul 30 17:02:53 2015 MULTI: multi_init called, r=256 v=256
Thu Jul 30 17:02:53 2015 IFCONFIG POOL: base=10.8.0.4 size=62, ipv6=0
Thu Jul 30 17:02:53 2015 IFCONFIG POOL LIST
Thu Jul 30 17:02:53 2015 Initialization Sequence Completed
The output of openvpn /etc/openvpn/client.conf on the client:
Thu Jul 30 21:03:02 2015 OpenVPN 2.3.6 x86_64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Dec 2 2014
Thu Jul 30 21:03:02 2015 library versions: OpenSSL 1.0.2d 9 Jul 2015, LZO 2.09
Thu Jul 30 21:03:02 2015 WARNING: file '/etc/openvpn/client-name.key' is group or others accessible
Thu Jul 30 21:03:02 2015 WARNING: file '/etc/openvpn/ta.key' is group or others accessible
Thu Jul 30 21:03:02 2015 Control Channel Authentication: using '/etc/openvpn/ta.key' as a OpenVPN static key file
Thu Jul 30 21:03:02 2015 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 21:03:02 2015 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 21:03:02 2015 Socket Buffers: R=[212992->131072] S=[212992->131072]
Thu Jul 30 21:03:02 2015 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Thu Jul 30 21:03:02 2015 UDPv4 link local: [undef]
Thu Jul 30 21:03:02 2015 UDPv4 link remote: [AF_INET][my public IP here]:1194
Thu Jul 30 21:04:02 2015 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Thu Jul 30 21:04:02 2015 TLS Error: TLS handshake failed
Thu Jul 30 21:04:02 2015 SIGUSR1[soft,tls-error] received, process restarting
Thu Jul 30 21:04:02 2015 Restart pause, 2 second(s)
Solution 1:
I had this problem as well.
Am using digitalocean provider for my server and the problem was with floating ip feature.
In order to fix that, you have to update openvpn config setting:
local <ip anchor>
ip anchor should be an ip adress gathered from ip addr command, see example:
Credits to this post
Solution 2:
As suggested by Michael Hampton and Michal Sokolowski in the comments on my question, it was a problem with the port forwarding rule I created on my gateway. OpenVPN is configured to use UDP, and I forgot to switch from TCP to UDP on the gateway as I usually don't use that protocol. The forwarding rule now uses UDP, and my VPN is functional.
Solution 3:
My current configuration would work on some countries but not others. I am suspecting that my current provider is blocking TLS handshake packet. Solution? Since I am the only one using that VPN I have switched to static key authentication which - in my case - proved to be super fast https://openvpn.net/index.php/open-source/documentation/miscellaneous/78-static-key-mini-howto.html