When to use HttpMessageHandler vs ActionFilter?

The major difference between their two is their focus. Message Handlers are applied to all HTTP requests. They perform the function of an HTTP intermediary. Filters apply only to requests that are dispatched to the particular controller/action where the filter is applied.

You should use MessageHandlers when you want the behaviour to be applied to the vast majority of requests. Filters should be used when they are only applicable to certain resources.

A big difference between handlers and action filters is the stage at which they are executed. Action filters are executed after controller dispatch and model binding has occurred, so you have the ability to interact with the controller instance that is handling the request, as well as having direct access to the typed model objects that are passed to the action method. I've used this approach to perform advanced logging of request/response values - because I could get at the controller, I was able to log additional information about how the request was handled.

Message handlers are executed earlier in the process, and operate much closer to the raw request/response values than filters (contrast the HttpRequestMessage/HttpResponseMessage objects used in handlers with the much-richer HttpActionContext and HttpActionExecutedContext objects available within filters). This makes them potentially more efficient for some activities, for example, if you're trying to determine whether you need to terminate the request early. If you know the request should be rejected, a handler will allow you to do this before the WebApi infrastructure has gone to the effort of instantiating the controller, performing model binding etc.

Another difference is that handlers are chained, and that you have more control over how the chain is executed. Although filters are called in sequence, setting the response in one filter will effectively end the request, preventing the next filter in the list from being executed. For example, if you have a first filter that returns a bad request if your model is null, and a second filter to log the request, your second filter would not be executed once the bad request is generated by the first filter. With a handler, it would still be possible (although not necessarily advisable!) to call the inner handler and allow the handler chain to operate normally.