Why was someone considered mentally unbalanced or crazy called a "crackpot"?

Solution 1:

OED's definition #5 for cracked...

Of the brain, mind, etc.: Unsound, impaired, somewhat deranged.
Of a person: Unsound in mind, slightly insane, crazy.

...for which the first relevant citation is...

1610 Bp. J. Hall Common Apol. against Brownists xxvii. 68
That which this man was wont so oft to obiect to his brother (a crack't braine).

As to the specific conjunction crackpot, I think it's worth noting that -pot is appended to various other "fused words" applied (usually, negatively) to people - moneypots, honeypot, fusspot, tosspot.

I think it's fairly obvious that crack't/cracked in this context is just a slightly more figurative way of saying broken, not functioning properly. Per @Oldcat's comment below, additional -pot component could be interpreted as head, brain, skull. Or it may be simply added for prosody (or by association with related forms), and not really add any specific extra meaning.

Solution 2:

FumbleFingers has a great answer for the history of crackpot. But know that I don't find this meaning to hold up with the younger generation.

If you call someone a crackhead, they are either an addict, or acting like a dumbass. My kids are definitely crackheads sometimes.

If you call someone a crackpot you are basically calling them a crackhead. Wrong or right the drug "crack" has changed the meaning to people.

Solution 3:

I think pot is being used in terms of "a collection of something" or another way to say "a lot of", so a moneypot is someone with a lot of money, a fusspot is a overly fussy person, a tosspot is someone who is very mucher a tosser (British slang) or very toss-y in nature.

A crackpot therefore, drawing on the suggested meaning for crack, would mean someone who is very crazy.