How do you get the gambling password in Monkey Island 2?

I'm not sure I remember correctly but I'll give it a try.

The correct answer is always the amount of fingers he shows in his example.

For example: If he says "If this is 3", and he holds up 4 fingers, then he holds up another number of fingers and asks "what is this?", the correct answer would be 4.

To make it clear, he shows you two sets of fingers and the number of fingers he holds up the first time is the correct answer.

So there is not really any formula or anything, and it doesn't matter what he says the first fingers correspond to, you just have to look at how many fingers he is holding up and that will be the answer to the next set of fingers.