What does the word "than" mean in the idiom "more often than not"?

It's the usual meaning of "than".

https://www.merriam-webster.com/dictionary/than

used as a function word to indicate the second member or the member taken as the point of departure in a comparison expressive of inequality —used with comparative adjectives and comparative adverbs

You seem to be confused by the phrase as a whole, not just "than". The phrase is used in reference to something that can happen or not. It's saying that the thing is more often than not the thing. In other words, it means that the thing happens more than half the time. MW gives the example of "He wins more often than not." This means that his winning happens more often than him not winning. It's short for "He wins more often than he does not win". That's really clunky because English requires the dummy "does", and it's simpler to just say "not".