why is "their" the wrong usage in this sentence?

Solution 1:

It's wrong because family is singular and their should be used for plurals (apart from singular their, which can't be the case here).

For singular you use its, therefore:

The modern American family differs in many significant ways from its nineteenth-century counterpart.

Or you make the subject plural, thus having:

Modern American families differ in many significant ways from their nineteenth-century counterparts.

Solution 2:

OP's sentence is wrong for most Americans because they normally, esp. in recent decades, treat "the family" as singular, so they would expect "its counterpart", not "their counterpart".

It's wrong for Brits because although it's true we often treat "the family" as plural, if we did that in this particular case we'd obviously expect "The ... family differ", not "differs".

But it's worth pointing out that Brits don't always treat these "group nouns" (family, Parliament, the government, company names, etc.) as plural. In this specific case I think most of us would use the singular anyway, because the family is referenced as a single entity being compared to another single entity (its "collective" counterpart).

It's also worth pointing out that I doubt many native speakers would be happy with...

?*The family took its places at the table.

...which just goes to show that there's no single "logical" approach anyway. I put that example up because you can fix it using singular "they", or simply by saying "the family" is plural anyway (in which case you can legitimately use "plural 'they'"). But if you insist "the family" is singular, and you don't endorse "singular 'they'", you're stuck with the very ungainly form above!