What is the proper HTTP response to send for requests that require SSL/TLS

Solution 1:

I cannot say if this is broadly accepted by HTTP clients, but speaking strictly RFC, the server should respond with:

HTTP/1.1 426 Upgrade Required
Upgrade: TLS/1.0, HTTP/1.1
Connection: Upgrade

Source:
https://www.rfc-editor.org/rfc/rfc2817#section-4.2

Solution 2:

The most secure way to force HTTP client to use HTTPS is HTTP Strict Transport Security.

Previously a common suggestion was to drop the connection, but this practice has been removed in favor of HSTS (OWASP website).

Solution 3:

The appropriate error code to return would be similar to 403.4 - SSL required.

Although not explicitly documented in the RFC for HTTP 1.1, this behavior does match the requirements outlined there:

The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity. If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead.

Adding your own subcode (as with the SSL example) might be helpful in some cases, but since this subcode would not be meaningful to third parties, I would recommend against it.

So, your final error message would be something like "403 - Private Resource". Note that, even in the case of a missing API key, "401 - Unauthorized" should not be used, unless your API key can actually be transmitted in a WWW-Authenticate header field.