Is "gaijin" considered offensive?

Firstly, for my first-hand reporting of the usage (and perceived offensiveness) I observed of this word, discussing with a small Japanese community in London and Boston. Gaijin is fine in normal conversation, both between non-Japanese people and between Japanese and non-Japanese.

Secondly, looking up some authorities, none of the dictionaries I have at hand or checked online (Merriam-Webster, Oxford, Cambridge) mark it as offensive or derogatory.

Thirdly, I thought about similar words for other communities. You gave the example of nigger, which is extremely offensive (“black” or “African American” being respectful alternatives). Another one is goy, which can be perceived as offensive depending on context and audience (and for which “Gentile” is a safer alternative).

In my experience, the Japanese seldom use gaijin to your face. If they are making an effort to be polite (which they almost always are), they will call, say, an American amerikajin or beikokujin (lit. "rice-land person"). If they're really being rude (it happens) they might use hakujin (lit. "white person"). If you hear that and you're Caucasian you may assume that you're being talked down to, although there's still a chance the rudeness is merely condescension or ignorance.

The politer form of gaijin is gaikokujin (lit. "out-country person").

Now, in English we make no such distinction. We just tag it as their word for foreigner, which glosses over all of the nuance.