Why is IE showing certificate popup when authentication mode in IIS is anonymous?

The browser is prompting for a certificate because the server sent a Certificate Request message during the TLS handshake.

Normally, as part of this, it sends a list of the Distinguished Names of CAs whose certificates it will accept for authentication. This list basically means "send me a client authentication certificate which was issued by one of these CAs". If it doesn't send a list, the client is at liberty to send any client authentication certificate it has, which may well not be trusted by the server. That would simply increase the workload/frustration of the operator who'd have to work out (guess?) which certificate to select.

Therefore you are seeing three certificates either because you only have three client authentication certificates installed in your browser and the list is empty; or you have more than three client authentication certificates installed in your browser and the list sent by the server restricts their selection.

Details are in RFC 5246 Section 7.4.4.

The option to request a certificate from the client is configured on the server, therefore you are presented with this popup because the server is configured to request a client certificate.

Remember that this happens before any HTTP traffic flows, so setting things like Anonymous Authentication on IIS will not affect this as that relates to how the user is authenticated over HTTP (none, Kerberos, username/password etc.) after the TLS session has been set-up.

Most webservers can be configured with different settings for different schemes (http vs https), different ports (443, vs 8443), different DNS name (www.example.org vs app.example.org) or even different virtual directories (/ vs /myapp). It is at this level where the setting to request client authentication is found.

In IIS client authentication is configured under the SSL Settings page. Require SSL sets whether a HTTPS is used (server identity and encryption) and the three options under Client certificates define whether the browser will be expected to send a client authentication certificate or not (you cannot have the latter without the former as client authentication is part of TLS (or SSL)). There is a different SSL Settings page for each site and for each virtual directory under a site. If you don't want the server to request client certificates, set this to Ignore in all of them.