Solution 1:

I have finally figured it out. I had to put my configuration in \.ebextensions\nginx\conf.d\elasticbeanstalk\*.conf, for instance \.ebextensions\nginx\conf.d\elasticbeanstalk\force-https.conf.

Contents are:

if ($http_x_forwarded_proto = 'http') {
    return 301 https://www.example.com$request_uri;
}

This gets included in the AWS ELB config file automatically:

# Elastic Beanstalk Nginx Configuration File
...
http {
    ...
    server {
        ...
        # Include the Elastic Beanstalk generated locations
        include conf.d/elasticbeanstalk/*.conf;
    }
}

So no need to copy/override the whole nginx.conf.