What is this type of idiom called: "I know he's not the smartest person in the world, but..."

It's called litotes.

From Wikipedia:

In rhetoric, litotes is a figure of speech wherein understatement is used to emphasize a point by stating a negative to further affirm a positive, often incorporating double negatives for effect

For example, "She's not bad looking" could be used to express that someone is gorgeous. Or it could convey that she's not particularly ugly, but also isn't particularly attractive.

The degree of emphasis depends on the context in which it is used.

From Oxford:

litotes: _{^{[MASS NOUN]}} Ironical understatement in which an affirmative is expressed by the negative of its contrary (e.g. I shan’t be sorry for I shall be glad).

I would call it a "euphemism", a figure of speech where you substitute a milder word or phrase for one considered to be too harsh or offensive. ("you know, he isn't the smartest person in the world" instead of "he is dumb".

• A euphemism is a generally innocuous word or expression used in place of one that may be found offensive or suggest something unpleasant. Some euphemisms are intended to amuse; while others use bland, inoffensive terms for things the user wishes to downplay. Euphemisms are used to refer to taboo topics (such as disability, sex, excretion, and death) in a polite way, or to mask profanity.

• euphemism - "a mild or pleasant word or phrase that is used instead of one that is unpleasant or offensive" MW

examples of euphemisms:

• to pass away instead to die
• ethnic cleansing instead of genocide
• pregnancy termination instead of abortion
• adult entertainment instead of pornography
• go all the way instead of have sex
• enhanced interrogation methods instead of torture
• undocumented workers (when they are working for you) instead of illegal aliens (when they are not)

You might call that a backhanded compliment:

A backhanded compliment, also known as a left-handed compliment or asteism, is an insult that is disguised as a compliment. Sometimes, a backhanded compliment may be inadvertent. However, the term usually connotes an intent to belittle or condescend.