Upgrade HTTP connection to SSL/TLS

A browser should never downgrade by itself to http if https does not work, because all an attacker would need to do is do make https unavailable (e.g. block port 443). So the only way this could be done is by instructing the browser from the server to do so, e.g. by sending a http redirect. Of course this should be sent over a secure connection (otherwise a man-in-the-middle could fake it) but unfortunately it is exactly your problem that the secure connection fails.

In summary: No, it is not possible and it is better this way.

BTW, all modern browsers support SNI, but not all applications (e.g. Java apps etc) do. If you have multiple domains on the web server but only a single one needed by these apps you could set the certificate for this domain it as the default. Otherwise you would need to get a (more expensive) certificate which contains all the needed domains as subject alternative names.

Edit with another idea: what you could try to do is to download an image from your side as https and check the success with an onerror handler on the img-tag. Maybe this does not trigger a user-visible warning but instead just fails to load. And if it succeeds you know that https access is possible and redirect the user.

Apart from that you have to ask yourself, why you want to offer https at all, if you accept the access with http too. Either there are data which should be protected or they are not. As long as you offer a fallback to http it is easy for an attacker to enforce http instead of https.

First of all it should be possible to specify one certificate to use for all clients lacking SNI support. This means of all the domains hosted on that IP address, you can have at least one of them working for clients without SNI.

What you could then do when redirecting from http to https is a two stage redirection. The first redirect from http to https uses the domain name, which you ensured would work with or without SNI support. The full original URL would have to be included, such that from this https site you can redirect to the proper one afterwards.

The domain name, which works with or without SNI can behave differently depending on whether SNI is supported by the client. That way you would know the client supported SNI before you redirected it to a domain, which required SNI.

How exactly to configure this on Apache is going to be a bit of guessing from my side (since I never configured Apache with more than one certificate). I guess the way to do it would be to create name based virtual hosts for all the domains including the intermediate domain.

Then create a default virtual host for clients without SNI, which uses the same certificate as the one using the name. Those two virtual hosts with identical certificate, will send different redirect to the clients depending on whether they support SNI.

Finally, I would enable IPv6 on the server. With IPv6 you get enough IP addresses that you can allocate one to each virtual host. The same set of virtual hosts can be name based on IPv4 and IP based on IPv6, so you don't have to duplicate any configuration this way.

The end result would be a setup which works as long as the client support either SNI or IPv6. Only clients which support neither will then have a problem, but you will still be able to detect those and server a different redirect or an error message.

As for clients that don't like the CA, my only suggestion is to recognize them by their user-agent and handle them how you seem appropriate. Make sure you do have a link to the https site, which they can click on in case you by mistake included too many clients.