What MIME type if JSON is being returned by a REST API?

The JSON spec suggests application/json, and that seems to be supported by the IETF and IANA registry.

On the second question, I think if the message handling fails in some way you should return a structured and standard error response as a JSON message; only if there is a failure to deliver the message to the backend handler for some reason should you consider an HTTP error code.

Update 2014-06-27: The days where clients (browsers) only worked with a 200 response are long past and the prevailing advice for RESTful APIs is to use HTTP response codes appropriate for the response, 2xx for successful responses (e.g. 201 Created for PUT; 204 No Content for DELETE) and 4xx and 5xx for all error conditions, including those from the API itself.

The MIME type of JSON is

application/json

http://www.ietf.org/rfc/rfc4627.txt

http://www.iana.org/assignments/media-types/application/

More specifically here:

http://www.ietf.org/rfc/rfc4627.txt