Ubuntu failover from Ethernet-to-ADSL-modem to USB 3G dongle

I have an internet connected Ubuntu server, connected by Ethernet to a residential ADSL line. This is adequate, most of the time, but I can't rely upon the ADSL link to be as reliable as I'd like. Murphy's law has already dictated that down-time comes when it is least convenient.

I'd like to use a 'pay-as-you-go' 3G USB dongle to provide a fail-over. I'm lacking information both about sample configurations that work for other people - and about the most suitable (inexpensive) hardware for my purposes. The objective is that, when my ADSL line is disconnected, the server detects this and dials up the 3G service - monitoring the ADSL line and switching back seamlessly when its connection is restored.

The principle service I care about is an OpenVPN tunnel - over UDP to a remote server. The other useful service would be Squid - so I'd still have web-access from my LAN when ADSL is disconnected.

Questions:

1. How difficult would this be to set up?
2. Would I be forced to 'roll my own' - or are there packages/howtos I've overlooked?
3. What (inexpensive) hardware is recommended (concerned with unattended reliability - mainly)?
4. Any anecdotes of success or failure from others who've tackled this already?

I have done this any times for customers and i have not found a developed system to make this so i have ever roll my own, the steps that Manwe has given you are more or less what i do when i need them but i will paste here any crude bash scripts i am using (when i have time i want to make this much better in python).

Basically i check if i have internet or not and if i am using the wan backup and make the changes needed

#!/bin/bash
PATH="/bin:/sbin:/usr/bin:/usr/sbin"
primary_gw="192.168.1.1" #for example.
check_one="8.8.8.8"
check_two="8.8.4.4"

#first we check internet connection.
if `ping -c 1 -W 1 $check_one |grep -E '(unreachable|100\%\ packet\ loss)' &> /dev/null` &&\
   `ping -c 1 -W 1 $check_two |grep -E '(unreachable|100\%\ packet\ loss)' &>/dev/null`
  then #if we don't have internet
    if [ -e /tmp/wan_backup ]
      #if we are using backup right now we try to change to primary connection
      then ./script_change_to_primary.sh && rm /tmp/wan_backup
      #else we change to wan backup.
      else ./script_change_to_backup.sh && touch /tmp/wan_backup
    fi
fi

#if we are using wan backup right now we check if primary connection works.
if [ -e /tmp/wan_backup ]
  then
    if `ip route add $check_one via $primary_gw; ip route add $check_two via $primary_gw;\
        sleep 2; ping -c 1 -W 1 $check_one | grep -E '(unreachable|100\%\ packet\ loss)' &> /dev/null &&\
        ping -c 1 -W 1 $check_two | grep -E '(unreachable|100\%\ packet\ loss)' &> /dev/null`
      then #don't works we clean the routes and stay using backup
        ip route del $check_one via $primary_gw
        ip route del $check_two via $primary_gw
      else #it works so we change active connection 
        ip route del $check_one via $primary_gw
        ip route del $check_two via $primary_gw
        ./script_change_to_primary.sh
        rm /tmp/wan_backup
    fi
fi

Given that you only want your server to use 3g if adsl goes down i would only use iptables snat or masquerade only in the adsl iface and i would block access to squid in ./script_change_to_secondary.sh, your files could be:

script_change_to_secondary.sh

#!/bin/bash
pon 3gIsp #this one it is going to change the default route of server anyway
#drop squid connections, you could disable here the boot snat or masquerade for adsl
#but given your adsl is not active i don't see the need anyway
iptables -I INPUT -s LAN_IP_RANGE -d SERVER_IP -p tcp --dport 3128 -j DROP

script_change_to_primary.sh

#!/bin/bash
poff 3gIsp
iptables -D INPUT -s LAN_IP_RANGE -d SERVER_IP -p tcp --dport 3128 -j DROP
/etc/init.d/openvpn restart

You should have too in /etc/ppp/ip-up.d/ a bash script with "/etc/init.d/openvpn restart", this way every time you connect to a ppp provider your openvpn will restart automatically.

Well like i have said it is a bit ugly and crude but it works :) if you find a integrated clean solution for this make me know please :), one good thing of roll-your-own it is that you have a complete control of the system, this is an oversimplification of what i do in any customers that have two or three connections alive at the same time and do a load balancing and QoS all integrated with scripts that detect connections problems and changes the routes and the QoS.

If you prefer an integrated solution to a roll-your-own you can use a distribution like zentyal, it supports what you want to use but it is a complete distribution tailored to create a SmallBusiness server, i usually prefer to configure my servers at my own but this is a good distribution that can be managed via web.

We have couple of smaller edimax 3g routers that have 1 wlan, 1 lan and usb for 3G stick. Lan port can be configured to be part of the lan or as primary wan port port (3g acting as failover). I'm not affiliated with edimax and we use them on the road or on conventions for their size (and one of the ones we have is battery powered).

Having said that I would go with a setup in the ubuntu server.

Answers:

1. Easy if you are experienced server admin. Otherwise if you are willing to spend the time to learn and understand what you are doing it's doable for a poweruser.
2. I don't know about available packages, but

change to 3g - script example:

pon YourIsp-name
route del default gw "your_adls_defaultGW"
route add -host "ping_or_other_test_host_for_checking_adsl_route" gw  "your_adls_defaultGW"
"do_other_stuff_like_restart_openvpn_maybe"

change to adsl - script example:

poff YourIsp-name
route add default gw "your_adls_defaultGW"
route del -host "ping_or_other_test_host_for_checking_adsl_route" gw  "your_adls_defaultGW"
"do_other_stuff_like_restart_openvpn_maybe"

Using dhcp with adsl will screw this simple example and you have to either modify /etc/resolv.conf with every change or use open dns servers like google 8.8.8.8 (or your own resolver). Now.. monitoring is a bit more difficult, but putting something like this to cron (don't, this is a naive exmaple), will give you failover to 3g and back.

ping -q -c 2 "ping_or_other_test_host_for_checking_adsl_route"  && TARGETUP=1 || TARGETUP=0 ;  echo $TARGETUP ; if [ "$TARGETUP" == 1 ] ; then ( rm /tmp/.adsl_down ; [ -e /tmp/.adsl_up ] || ( PATH_TO_YOUR_ADSL_UP_SCRIPT && touch /tmp/.adsl_up ) ); else  ( rm /tmp/.adsl_up ; [ -e /tmp/.adsl_down ] || ( PATH_TO_YOUR_3G_UP_SCRIPT && touch /tmp/.adsl_down )) ; fi

Three) See Edimax 3g routers ( zeroshell linuxfw-distribution looks like zeroshell has failover in it, so get a cheap pc for a router and use zeroshell. I haven't tested it though )

Four) Test, persist and when failover is needed for the first time, it's going to fail (because of something stupid...).

I'm not sure that you need squid for this setup unless to speed up 3g connection with local cache of most common pages you visit..