WHY do so many people struggle with ‘who’ and ‘whom’?

I think there are a few reasons:

• Most people are not great at taking an explicit grammar rule and just adopting it; rather, we're much better at internalizing rules when we also have exposure to language that conforms to those rules. Since whom is rarely and inconsistently used, most people don't have enough exposure to it to get a good sense of when it's used.

• Most people (including most English teachers, most popular grammar and style writers, etc.) are not great at formulating explicit grammar rules, partly because they don't give a coherent overarching grammatical framework that those rules can fit into. Normally that doesn't make much difference because the explicit grammar rules aren't really how you learn grammar, but with something like whom where explicit grammar rules are almost all you've got, this is a problem.

  • In the specific case of whom, a large part of the problem is that it often sits at the intersection of two clauses, yet explanations of it never seem to worry about that.

• There's a close relationship between who/whom and certain other areas where traditional grammar differs from everyday English:

  • When to use subject vs. object pronouns. Do we say "It is me", or "It is I"? "Me and Jamie", or "Jamie and me", or "Jamie and I"? "She is taller than him", or "She is taller than I"?

  • Preposition stranding vs. pied piping. Do we say "that we spoke of", or "of which we spoke"?

  
  So when trying to understand the grammar of whom, we also have to balance all the other pieces of formal grammar that we don't usually worry about.

• The grammar of whom is often somewhat "long-range", in that the pronoun can be separated from the verb or preposition that it's the subject or object of. Consider this bit from Daniel Defoe's 1719 novel Robinson Crusoe:

  This ſeaſon I found my family to encreaſe; for one of my cats that ran away from me, and whom I thought had been dead, returned about Auguſt, with three kittens at her heels, like herſelf, which I thought ſtrange, because […] [link]

  where whom is erroneously being used as the subject of had, apparently because the writer (or narrator) felt it to be the object of I thought. You'll see this sort of mistake even with points of grammar that are not common sources of confusion; you'll encounter things like "Talking to people you don't know, about things you don't understand, sometimes make you look foolish" [made-up example], where make should be makes, but where the singular-ness of talking has become less salient by the time the speaker got to the verb.

I think a relevant factor is the much more rare use of "whom" compared to "who" in common speech. According to an article in The Economist,

A search of the Spoken category of the Corpus of Contemporary American English finds that I is about eight times more common than me—but who is 57 times more common than whom.

• Is "whom" history? From the mouths of babes: How is a child to learn "whom" when adults hardly use it?

The article points out the difficulty for English learners, particularly young children learning native English, of learning the appropriateness of a word that is rarely used at all by adults in speech.

It has even been suggested by serious linguists that "whom" will someday be as obsolete as "thee" or "thine;" lost through the same process of obsolescence that keeps the English language elastic and constantly moving.

When a word is rarely used in casual speech, it becomes subject to conflict between prescriptivist grammarians and descriptivist linguists observing the evolution of the language. In the case of "whom," this means that students are taught to use the word correctly in writing, but in casual speech it is often foregone for "who."

For example, in writing, I would probably have the discipline to write:

To whom did you give the book?

But when speaking with fellow native English speakers, I would almost certainly say

Who did you give the book to?

It wouldn't surprise me if many masters of the English language follow the same habits in common speech. Since speech is often how English is learned, particularly by children, it seems natural that they would struggle to master a rule that is oft violated by the native adult authorities themselves.