Disable HTTPS for a domain that shares IP with another domain that is under HTTPS

You won't be able to do what you want. You either:

1. Need to obtain a valid certificate for both domains and use the serverHost directives to redirect off SSL (to http://) if you truly don't want to run SSL on that domain.
2. Need to obtain a second IP address and run the domains on seperate IP addresses.
3. Leave SSL Enabled and serve an invalid certificate for domain1

WHY:

1. A client types https://domain1.com/ in a browser
2. The browser resolves this to a TCP connection to on port 443
3. The magic of the internet routes this connection to your server and hopefully into your web server software
4. The client browser is standing at the front door waiting for the proper 'secret handshake' (SSL Cert.)
5. Your server tries the only handshake it knows (which is wrong)
6. The client's browser informs the client that "This guy at the door of domain1.com... He's not who he says he is. This is sketchy"
7. Today, will click the variant of "Proceed Anyway" that their browser presents
8. IF you disable SSL on domain1.com, the user will be shown the other site (quite confusing for the user)
9. IF you leave SSL enabled on domain1.com, modern browsers and webservers will communicate and show the correct site. (albeit, with a scary warning)

If you intend to host domain1.org @ ip on port 80 and domain2.org at ip on port 443 (so that domain2 does not have HTTP access, it should 'work'). Remove the 443 listener for domain1. You will be prompted if any audacious users type https://domain1.org (and then shown domain2). That setup is essentially Port based virtual hosting.Instead of picking random ports you have picked the two defaults and given them to separate virtual hosts. Ultimately in this setup (WITH SSL laying around) the webserver doesn't really care what domain the browser is asking for in the Host header. The web server just cares about the ip:port combo in port based virtual hosts.

That's a quite common problem with http and https sites and not only a Nginx issue. You can have the same with Apache, IIS ... and whatever.

How to solve this correctly

Acquire an additional IP address for your Web server and address domain2.org to the new IP (IP #2). Additionally take care that your Web server is listening only on port 80 via IP #1 and is listening on ports 80 and 443 via IP #2.

Work-around if second IP not possible

If an additional IP address is not possible you can only implement a work-around. Activate SSL/TLS for domain1.org with the certificate of domain2.org and provide a redirect to http://domain1.org/ as you already did. Users will get SSL/TLS warnings anyway and most of them ignore these warnings. But so they get forwarded to the correct URL.

Example configuration for nginx:

ssl_certificate                 /etc/ssl/certs/domain2.org.pem;
ssl_certificate_key             /etc/ssl/private/domain2.org.key;
ssl_protocols                   SSLv3 TLSv1;
ssl_prefer_server_ciphers       on;
ssl_ciphers                     AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH;
ssl_session_cache               shared:SSL:10m;

# domain1.org
server {
    listen          80;
    listen          [::]:80;
    server_name     domain1.org;
    # your configuration part ...
}
server {
    listen          443 ssl spdy;
    listen          [::]:443 ssl spdy;
    server_name     domain1.org;
    return          301 http://domain1.org/;  # enforce correct protocol
}

# domain2.org
server {
    listen          80;
    listen          [::]:80;
    server_name     domain2.org;
    return          301 https://domain2.org/;  # enforce correct protocol
}
server {
    listen          443 default_server ssl spdy;
    listen          [::]:443 default_server ssl spdy;
    server_name     domain2.org;
    # your configuration part ...
}