Expressing an opinion: to me or for me?

I think the general (most widely-applicable) one would be "to me". To my knowledge, it can be used in any of these cases where "for me" is used, and in some where "for me" can't be used.

For example:

• For me, this is not a difficult problem.

• To me, this is not a difficult problem.

Both of the above are fine.

But look at the following pair (the "*" sentence is ungrammatical in this context):

• *For me, he is an idiot.

• To me, he is an idiot.

The first example doesn't work as a way to express opinion, but "to me" still works fine.

"For me" is to express its effect on you or it's benefit for you, whether it's good or bad.

"To me" is more to express opinion.

Ex: That's difficult for me. That sounds difficult to me.

Hmm, tough one. Rearranging the sentences helped me see this a little more clearly:

It makes no difference to me

This suggests there is no effect on me materially, emotionally, financially ...etc. That is, nothing will happen to me.

It makes no difference for me

This suggests that I have no strong opinion on this.

I can't explain this in terms of grammar or style, it's just a feeling that the sentences provoked when I said them to myself.