Adjusting how long Linux takes to fail over to backup DNS server listed in resolv.conf

Currently I'm using the plain vanilla linux configuration for resolv.conf... something like:

nameserver 123.123.123.123
nameserver 8.8.8.8

When 123.123.123.123 goes down DNS queries become impossible slow, I'm assuming that Linux retries the first one each time. Is there a way to get linux to be smarter about this? Health checks or something? Or do I misunderstand how resolv.conf should work?

In addition to ewwhite's awesome response, some addendum.

You can add this in /etc/resolv.conf

options timeout:1 attempts:1 rotate

The defaults are time:5 attempts:2

What happens is that the resolver library will try to use the nameservers listed in /etc/resolv.conf from top to bottom if no rotate option is present. If rotate is present, then it does a round-robin selection. If the resolver goes to the bottom of the list and the server doesn't respond within X seconds (considering X is the timeout parameter) then it will repeat the whole process of round robin selection again for Y-1 times (where Y is the value of attempts).

However a bit of caution would be to avoid dig and friends for testing these resolv.conf options. As they avoid resolver library and directly ask the nameservers. getent hosts is the better command to use. Note that anything that uses glibc resolver will have to obey /etc/resolv.conf file.

This is described via man resolv.conf.

Please try one of the timeout options to lower the default from 5 seconds to perhaps 1 second...

options timeout:1
nameserver 123.123.123.123
nameserver 8.8.8.8

But really, DNS has so many resiliencies available, it's possible to live without lower resolution timeouts. Is it possible to pick better public DNS or run your own internal resolver?