AWS Route 53 Failover DNS with Healthcheck not updating IP (even though health check shows failure)

Solution 1:

From the configuration shown in the question it looks like there is only one health check tied to both the primary and the secondary records?
In that case both the primary and secondary record will always be considered up/down at the same time, which undermines whole idea of selecting the best one which is up.

Create a health check for each endpoint and tie them to the corresponding records.