Why do I have a different vowel in "scarf" than I have in "scarves", and how come nobody talks about this?

No, you aren’t crazy; you have a really good ear. And as a native speaker, you can’t have been pronouncing those “wrong” all these years. Your tart simply has a very slightly different vowel than your tars.

It’s normal to have a “higher” vowel when there’s an unvoiced consonant after it, particularly in North America. What you’re observing is the same raising that happens in writer compared with rider: the main /ɑ/ vowel is raised to /ʌ/ as in cut.

That means scarf comes out as [skʌɹf] but scarves comes out as [skɑɹvz]. It may actually be [ɐɹ] rather than [ʌɹ], but that’s a very technical distinction that you probably won’t be familiar with.

Neither of those has an /o/ sound like wore/war (both either [wɔɹ] or [woɹ]) has, or for that matter like dwarf [dwoɹf] does. Oddly, the plural dwarves [dwoɹvz] is unchanged in its vowel. That’s curious.