in-able ? un-able?

Suffix -able adds meaning "being able" to a word. I know that.

Prefix in- and un- mean "not" or some negative meaning. I know that.

However, when it comes to mixing of these, I am confused.

• unbelievable
• indispensable

In the dictionary I can find many words of these forms, let's call them in-able and un-able, whose composing rule seems just random to me.

When given a certain word body, how can I make an in-able or un-able word to mean "not being able"?

Consider unbelievable versus its synonym incredible, and you will find what there is to be found of an answer here.

The general tendency is to use un‑ on Germanic words or any generic English term, and to use in‑ (possibly mutated; see next paragraph) for words of Latin origin. This is not hard and fast, however; there are of course exceptions.

Also, the Latinate in‑ prefix comes in other forms, like illegible, immutable, irreducible. Those also count as in‑ versions, not un‑ versions.

Similarly, ‑able is the more general ending, but -ible also frequently occurs. In fact, you will find that ‑able and ‑ance tend to go together, just as ‑ible and ‑ence tend to go together.

Here again there is an etymological explanation: whether it derived from either a Germanic word or from a first-conjugation Latin verb (so Germanic words or Latin ‑are verbs yield ‑able type endings) on the one hand, or whether it was instead from another Latin conjugation (so ‑ere verbs and such yield ‑ible type endings) on the other.

But as before, there are notable exceptions.

I don't think there is a rule for that. Both preffixes un- and in- usually apply to adjectives, and that is what you get when you have something-able.

The only "rule" I can remember is that for some verbs, when you want to mean the reverse action, un- is what you want (undo, unlock, untie...).

I agree with elias that there is no simple rule for this. Something that supports this viewpoint is that a number of words have shown variation over time, or still show variation: a famous example is "inalienable" vs. "unalienable".

But it may be possible to give fairly accurate rules for certain subcategories of -able adjectives.

If the -able word is based on a monosyllabic verb, use un-.

If you can remove the suffix -able from the adjective and get a monosyllabic English verb, the adjective almost certainly is negated with un- and not with in-.

I would guess that there are more than a hundred examples of words that follow this rule: to start with, you can consider unthinkable, unspeakable, unbearable, unstoppable, unflappable, unshakable, unforeseeable, unsalable, unquenchable, unbridgeable, unworkable, unlovable, unlikable, unwearable, unbreakable, unchainable, unplayable, unwinnable.

Un- is possible even when the verb is of French or Latin origin, as in untouchable, untreatable, unusable, unnotable.

I only know of four common exceptions to the monosyllable rule: incurable and impassable (for which the un- alternatives have negligible usage), immovable (which is much more common than unmov(e)able), and insolvable (which is currently less frequent than unsolvable). H. W. Fowler's Modern English Usage (first published 1926; republished 2009) mentions these four in a longer list of in-_-able words (-able, p. 5; for more information about Fowler's list, go to the next section).

Insuitable in place of unsuitable seems to be obsolete, although it can be found in dictionaries and some old documents.

If the -able word is not based on a monosyllabic verb...

An -able adjective that is related to a verb of more than one syllable may take either un- or in- as the negative prefix: un- is generally more common and productive, but I don't know of any particularly simple rule that tells you which prefix to use for all words in this category. You can try to reason from the etymology, as described in tchrist's answer: an -able word built on a verb with Germanic etymology will take un- as a rule. The etymological criterion can be used to rule out in- for unanswerable, unutterable, unforgettable.

However, etymology usually can't be used to rule out un-, because un- is used with many -able words derived from Latin or French.

Fowler (1926, cited in the previous section) gives a list of around 100 -able words where he recommends forming the negative with in-, and says to use un- for any word not on his list.

Minor rules based on the spelling of the end of the word

• If the word ends in -kable, use un-.

  I don't know of any exceptions to this in modern usage, but the words covered by it are mainly a subset of the words covered by the previous rule. We do also get unremarkable, unmistakable, unrebukable, unattackable. Inattackable seems to have once existed, but I think it's pretty much never used anymore.

• If the word ends in -cable, you can use in-.

  There are words ending in -cable that can be negated by un- (e.g. uneducable, although apparently some people prefer the sound of ineducable). But I haven't found any -cable adjectives that cannot be negated with in-.

• If the word ends in -yable, use un-.

  As with -kable, most examples are monosyllables, but we also have undestroyable, unemployable and unenjoyable. In-/im- may be found in unassimilated French words that are occasionally used in English such as impayable, incroyable.

• If the word ends in -onable, use un-.

  There aren't so many words that end like this, but I think there are enough to identify this as a pattern. Many of these words end more specifically in -ionable or -tionable. Examples: unquestionable, unexceptionable, unmentionable, unobjectionable; unconscionable, unfashionable; unseasonable, unreasonable, unpardonable. "In-" is not always completely impossible in this context; "inconscionable" exists, but is much less common. "Infashionable" has been used occasionally in the past (it's in the OED), but is now obsolete.