What happens if a resolver encounters a DNSSEC algorithm it does not support?

Solution 1:

What happens if a resolver incounters a DNSSEC algorrithm it does not support?

DNSSEC is built to allow validation as long as ONE validation path exists. If the resolver is able to find out at least one algorithm it supports and check that everything (signatures, etc.) are ok, then the DNSSEC validation will be ok, and the unknown algorithm will be ignored.

Of course, if the resolver finds only algorithms it doesn't support and not even one it supports, then the DNSSEC validation will be failed, and SERVFAIL will be returned.

See §5.3.1 of RFC 4035:

It is possible for more than one DNSKEY RR to match the conditions above. In this case, the validator cannot predetermine which DNSKEY RR to use to authenticate the signature, and it MUST try each matching DNSKEY RR until either the signature is validated or the validator has run out of matching public keys to try.

So one good algorithm is enough, the resolver does not have to test all of them. Which is good to be able to introduce progressively new algorithms at the publishing side, while resolvers are updated to know about the new algorithm.

Have a look also at RFC 8626 "Algorithm Implementation Requirements and Usage Guidance for DNSSEC".

Does it refuse to return the requested record or does it return the record, treating the domain as unsecured?

DNSSEC is binary normally: the domain has correct DNSSEC validation or is in error (hence SERVFAIL). A resolver is not allowed to strip DNSSEC out of a domain if it failed, and return records as if the domain didn't have DNSSEC from the beginning.

However, that is the theory. In practice, it does happen that recursive resolver need sometimes to continue answering even for domains known to be DNSSEC broken because it is considered that it will create otherwise a too big burgen. See the famous past NASA/Comcast example. Which is why there are now "Negative Trust Anchors" or NTAs: this is a way for operators of recursive resolvers to say "domain X is DNSSEC broken, ignore failing trust there and operate as if there is no DNSSEC". It is supposed to be a temporary measure, and is obviously local to each recursive resolver.

See RFC 7646 "Definition and Use of DNSSEC Negative Trust Anchors" (september 2015):

This document defines Negative Trust Anchors (NTAs), which can be used to mitigate DNSSEC validation failures by disabling DNSSEC validation at specified domains.