Both their three cars
Both indicates that the action or state denoted by the verb applies individually to each of two entities. Both books weigh more than five pounds means that each weighs more than five by itself, not that the two books weighed together come to more than five.
https://www.ahdictionary.com/word/search.html?q=both
Is both their three cars ambiguous with the meaning "each owns three cars, six cars in total"?
What about the three cars of both('s?) or both's three cars?
BOTH (OED) https://www.oed.com/oed2/00025622
Solution 1:
"Both their three cars" sounds really wrong because "both" typically modifies the noun following it: "both" modifies "books" in the sentence from ahdictionary. In the phrase "both their three cars" at first glance it looks as if "both" is modifying "cars" but then it is specified that there are three cars, making the phrasing confusing. To indicate that the "both" is modifying two people referred to previously, you would need to use "they both".
For example, instead of "Both their three cars needed an oil change," you could use "They both had three cars in need of an oil change" or "They both had three cars that needed an oil change."
Solution 2:
The word both requires the referent of the noun phrase it combines with to be exactly two. So when you say:
- I like both of the cars.
- I like both (of) their cars.
the sentence is only felicitous in a context in which there are only two relevant cars.
So sentences like your example is simply ungrammatical, since the 'three' in the noun phrase is incompatible with the requirement that the noun phrase refers to two things.
- *I like both of the three cars.
- *I like both (of) their three cars.
With respect to your second pair of examples (which as given aren't actually grammatical) we can construct examples like:
- The three cars of both of them
- The three cars of them both.
These examples aren't that great, but my interpretation of them is that they mean that each person has three cars, so six cars in total. I'm not sure if there is an interpretation in which there are three cars which belong to them both collectively.