I'd rather not [do something] vs I'd prefer not to [do something]

In this question the issue came up as to whether there's any difference in the level of politeness/correctness involved in I'd rather not say as opposed to I'd prefer not to say.

My own gut feeling is the prefer form is a bit more "formal" (and thus arguably more correct/polite). This NGram shows that I'd rather occurs far more often than I'd prefer. Partly that's just because the rather form is more common, but the difference is far less marked with the more formal I would rather/prefer. I take this to mean rather is more suited to casual use/conversation when used with this meaning.

It could be lots of verbs besides say, and I don't think the not is necessarily relevant here either. But usages vary for certain alternatives - for example, go blind only seems to work with rather.

That's just my opinion. Does anyone have arguments/evidence/sources to either back up or refute the proposition that I'd prefer not to say is more formal than I'd rather not say?

Solution 1:

In my experience, "rather" is stronger - and therefore less polite - than "prefer," unless "prefer" is accompanied by a snooty tone. "I'd prefer not to do that" definitely sounds not only more polite to my ears, but also more willing to do the thing if necessary. "I'd rather not do that" seems to carry a more heavy reluctance, and to be a bit more casual as well.

n.b. I am a native speaker of English in the USA.