I have an nginx and different subdomains:

a.mydomain.com
b.mydomain.com
c.mydomain.com

Nginx has 4 rules:

1) rewrite rule:

server {
  listen 80
  server_name gl.udesk.org;

  root /nowhere;
  rewrite ^ https://a.mydomain.com$request_uri permanent;
}

2) https rule:

server {

  listen 443;
  server_name a.mydomain.com;

  root /home/a/a/public;

  ssl on;
  ssl_certificate conf.d/ssl/a.crt;
  ssl_certificate_key conf.d/ssl/a.key;
  ssl_protocols ...
  ssl_ciphers ...
  ssl_prefer_server_ciphers on;

  location ...
}

3) http default rule:

server {
  listen 80 default_server;
  return 444;
}

4) https default rule:

server {
  listen 443 default_server;
  return 444;
}

So if I start nginx and:

• if I go in the browser to http://a.mydomain.com it redirects to https://a.mydomain.com and then it returns an Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.
• if I go in the browser to https://b.mydomain.com I expect that it returns Error 444 back. But instead it returns the same Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.
• and so for all registered by the DNS provider CNAMEs (i.e. a,b,c)
• all the http-versions (e.g. rule 3 - ) working as expected:
  • http://a.mydomain.com redirects to the https:// version,
  • http://b.mydomain.com and http://c.mydomain.com are returning an Error 444 back as configured.

So why the https rules in nginx are so tricky to configure and how should I configure them properly to get the same behavior as with http version?

Update:

Creating a new certificate and adding:

ssl on;
ssl_certificate conf.d/ssl/default.crt;
ssl_certificate_key conf.d/ssl/default.key;

works now, but I would have a solution without any SSL certificate needed. Just reset all connections for all https (port 443) subdomains except https://a.mydomain.com without providing a certificate.

Don't mix Port 443 with ssl! Nginx is completely port agnostic. You can offer https through Port 80 too. Modern nginx versions allow

listen 1234 ssl;

and you don't need the ssl on; line then.

But if you want to serve https you need to specify a certificate. Your server enters https when it rewrites the http request into a https request.

You get the PROTOCOL ERROR, as the SSL Handshake is done before anything else. So return 444 isn't reached. And any SSL Handshake will need a ceritificate and a private key, to feed the encryption algorithms with the certificate/private key pair.

The return directive is part of the rewrite module. If you check the documentation, you may see that it works with requests. In HTTPS requests can only be made after the handshake has been finished.

There's a feature request: https://trac.nginx.org/nginx/ticket/195 and a workaround solution is provided.

server {
    listen 443 ssl;
    server_name bbb.example.com;
    ssl_ciphers aNULL;
    ssl_certificate /path/to/dummy.crt;
    ssl_certificate_key /path/to/dummy.key;
    return 444;
}

I found answer in Nginx docs:

The SSL connection is established before the browser sends an HTTP request and nginx does not know the name of the requested server. Therefore, it may only offer the default server’s certificate.

Thus, you must configure the default certificate for SSL connections to default_server that will use for request first. After that nginx will search for the best match server_name virtual server and use its config.

Also you may need to set the SSL connection settings for the default_server, even with SNI support, because some SSL settings in matched server config will be ignored. I don't understand how it works yet, but nginx receives to client ssl_certificate from matched server config and ssl_protocols from default_server config.

And now, since Nginx 1.19.4, you can use ssl_reject_handshake directive:

server {
  listen               443 ssl;
  ssl_reject_handshake on;
}