Getting correct SERVER_PORT to php-fpm through nginx and varnish

Found my own answer

In the SSL nginx, this...

   proxy_set_header X-Forwarded-Proto https;

then in varnish, this...

 sub vcl_recv {
   if (req.http.X-Forwarded-Proto == "https" ) {
     set req.http.X-Forwarded-Port = "443";
   } else {
     set req.http.X-Forwarded-Port = "80";
     set req.http.X-Forwarded-Proto = "http";
   }
 }

then in the final nginx, this...

fastcgi_param  SERVER_PORT        $http_x_forwarded_port;