What determines whether the FQDN is interpreted as IPv6 or IPv4?

As this is tagged iptables, I assume this is a Linux question.

On Linux it is up to glibc to decide whether a program, for a given hostname, looks up the IPv6 address (AAAA) or the IPv4 address (A) first. The program using glibc uses the getaddrinfo() function. The behaviour of getaddrinfo() is configured in /etc/gai.conf.

On your system, most likely everything in /etc/gai.conf is commented out. Uncommenting the entry:

label ::/0 1

has, for me, led to the activation of AAAA-first lookup, so IPv6 addresses get looked up preferredly. So this may be what you want.

Update to add: The proper way appears to be to uncomment the entire "label" block in gai.conf with the values from 0 to 7:

label ::1/128       0
label ::/0          1
label 2002::/16     2
label ::/96         3
label ::ffff:0:0/96 4
label fec0::/10     5
label fc00::/7      6
label 2001:0::/32   7

IPv4 addresses are mapped to a domain name by an A (Address) record in DNS (Domain Name System). IPv6 addresses are mapped using AAAA Records. Seems strange? IPv4 addresses are 32bits, IPv6 are 128bits. 128 / 32 = 4, So AAAA / A = 4 as well. Clever trick by whoever designed IPv6's entry in DNS.

DNS is a hierarchical, distributed system across the whole of the internet. Your browser will make a call to a system library, which starts the lookup chain. It'll check its local cache first, usually stored in memory. Next, it seeks to disk and looks at the HOSTS file (/etc/hosts on most UNIXes/Linux, and C:\Windows\System32\drivers\etc\HOSTS on Windows). Then, if it still can't find an answer, it makes a call to your assigned DNS server, per your networking configuration.

THAT DNS server checks its cache, which is much larger. If it has a match, it returns an answer. If it can't find an answer, it turns around and asks an even higher-level DNS server run by the ISP of the ISP. This process continues until you hit the root name servers, a.root-servers.net through m.root-servers.net. A majority of DNS queries are resolved LONG before they get that far, but sometimes one or two make it up there. But when an answer is found, it's passed right back down that chain to your browser.

But how does your browser know which record type to request? That depends on your system settings. If you've set it to your IPv4 stack is priority, then your browser will only ask for A records in its DNS queries. If IPv6, it'll ask for an AAAA record first, and then fall back to an A record second.

There are other record types, too. MX records define where the mail server for a domain is. NS records define the name servers for a domain. SRV records tell you where a particular service (SSH or Web, for example) are at.

DNS really is the literal phone book of the internet.

The lookup would be handled by your DNS (and possibly your local hosts file). Whether a particular name is resolved as an IPv6 or IPv4 address depends on the settings of your browser (does it support IPv6 and does it send IPv6 AAAA requests, your operating system (whether you even have an IPv6 stack amongst other things), and your DNS provider (do they support AAAA (IPv6) requests or not).

The browser issue is complicated further by the fact that some (older) browsers never issue AAAA lookup requests and some can be configured not to send them (firefox has a setting for this) but most modern browsers send an AAAA request first and follow up with an A request.