"Exchanged with" vs. "exchanged for"

Is "exchanged with" grammatically correct and does it mean the same thing as "exchanged for?" "For" and "with" don't normally seem interchangeable, so these two phrases should be different, yet they seem similar.

Exchange with a person you are transacting with

I exchanged wares with the baker today.

Exchange a thing you spent for a thing you received as in a purchase or scheme

I exchanged a sack of flour for a rhubarb pie.

Exchange one thing with another if you control both of them (less frequent)

It wouldn't all fit in my bag, until I rearranged the things in my bag by exchanging the pie with the dictionary.

It may be that for some speakers, and/or in some contexts, the two expressions are interchangeable, but I don't think that applies to most usages.

Exchanged with is relatively uncommon. It's used when sub-elements of a system under consideration are being altered for some reason (optimisation, analysis, etc.) And in the sense of exchanged correspondence/contracts/currency/etc. with.

Exchanged for, the more common form, generally implies a transaction (barter, swap, whatever) between two parties. Normally in contexts where one or both 'exchangers' consider the exchange to be at the very least 'fair', if not actually 'advantageous'. These are not relevant issues in the contexts outlined above where with is normally used.

LATER: It's been pointed out that there are specific contexts where the two usages would be considered different by almost all speakers, even if they wouldn't conciously acknowledge or adhere to the distinction given above.

Here is an NGram specifically graphing written usage of exchanged for him compared to exchanged with him. If you leaf through some of them, you'll see that the vast majority of references to with are in the specific contexts specified above, even though these contexts represent a tiny proportion of all references to 'exchanging' in general.