Cambridge records both meanings under the same meaning:

to not feel certain or confident about something or to think that something is not probable:

  • I doubt whether/if I can finish the work on time.
  • [ + that ] They had begun to doubt that it could be done.

But looking at the meaning of each sentence, I would interpret them as:

(a) I doubt/ [wonder] whether he'll do it. (It is likely he will do it, but I am not sure)
(b) I doubt/ [don't think] that he'll do it. (I think it is unlikely he will do it)

This site agrees:

If you really doubt that something is true (suspect that it’s false), use “doubt that”:

  • I doubt that Fred has really lost 25 pounds.

If you want to express uncertainty, use “whether”:

  • I doubt whether we’ll see the comet if the clouds don’t clear soon.”

Doubt if” can be substituted for “doubt whether,” though it’s considered somewhat more casual, but don’t use it when you mean “doubt that.”

Here is what Fowler says [H.W. Fowler (1858–1933). The King’s English, 2nd ed. 1908]:

The verb 'doubt', which is constructed with 'that' or 'whether' according to the circumstances under which the doubt is expressed, requires special notice.

  1. If there is nothing to show that the writer considers the doubt an unreasonable one, the word is always 'whether', which reminds us that there is a suppressed alternative:
  • I doubt whether this is true (or not).
  1. If it is evident that the writer disapproves of the doubt, the words introducing it amount to an affirmation on his part that the thing doubted is undoubtedly true; the alternative is no longer offered; 'that' is therefore the word:
  • I do not doubt that (i. e., I am sure that)...

Fowler goes on to examine some exceptions, it's an interesting read. The conclusion is that doubt that has rather a degree of certainty, whereas doubt whether has a higher degree of uncertainty.