Is “flatter than a mashed cat” a common phrase?

As others have pointed out, it's not a popular phrase.

Why a cat? Why not a possum, or a skunk?

When I read the metaphor, I immediately assumed it was a pun on the word "flat," and an allusion to roadkill (in other words, "You're flatter than a run-over animal!" - which doesn't read nearly as vivid as the original "mashed cat").

No way to tell for sure, I guess, unless Mr. Carhart frequents this forum.

I have never heard this phrase before. I compared it to another phrase regarding thickness, and the results show that it's not very popular as compared to flat as a pancake:

After that, I checked to see if it had any results:

Nope. I would say that it's safe to say that this is not a common phrase.

I also tried Googling things like "flatter than a mashed" (that one turned up four results) and "flatter than a mashed cat" (which turned up 2 results), so I can't find any related phrases.

To scream/squall like a mashed cat is a pretty rare turn of phrase, but it accounts for most of the 75 instances of "mashed cat" in Google Books.

Of those 75, the only one I see making any reference to said cat being "flat" is OP's citation. Almost all the others refer to it being angry/howling. Perhaps Carhart chose "cat" because he'd heard "mashed cat" before, in that different context.

It makes little difference whether Carhart knew his usage was "non-standard" for the term - he's a creative writer. As @Jim says, "flat" here primarily means off-key (singing at too low a pitch), but with the obvious allusion to a "squashed" cat being physically flat (as a pancake).

Possibly he intended an oblique comparison between an off-key singer and a screaming/yowling "mashed cat". It's also possible he's alluding to "cats" as jazz musicians from a generation or two ago, and to "mashed" in the modern sense of stoned, intoxicated (a musician in that state might well play flat). But we're getting into realms of Lit. Crit. there, beyond the scope of ELU.