Server hello is not returned without SNI
SNI is a technology specifically designed to allow a server to host multiple sites with different certificates on a single IP address. in order to do that though, the HTTP request must contain a host
header.
Generally when a webserver admin sets up SNI, they select a default host to use when no host header is provided, as would be the case with a request by IP address. the details depend on the webserver in question, but you should be able to find instructions for whatever webserver you are running.
Note that this issue impacts servers configured to use SNI regardless of your proxy implementation. the issue and the fix are not directly related. there are just some sites you can't use by IP.