If you do not need load balancing, track scripts offer failover based on checks run against your service.

First, add a vrrp_script block before your vrrp_instance:

global_defs {
    enable_script_security
}

vrrp_script chk_sshd {
    script "/usr/bin/pgrep sshd" # or "nc -zv localhost 22"
    interval 5                   # default: 1s
}

Next, add a track_script to your vrrp_instance referencing the vrrp_script:

 vrrp_instance VI_1 {
    ... other stuff ...

    track_script {
        chk_sshd
    }
}

While not strictly required, the enable_script_security and FQDN of the executable provide some assurances against malicious activity and will squelch warnings in logs. See the Keepalived man page for more info.