Find two positive real numbers, whose difference is 100 and whose product is a minimum

If the minimum of a function lies on the boundary of the region of parameters, then simple differentiation will not work. You must also check the boundaries, which in this case includes the (true) solution $(0,100)$.

A trivial illustration: Find the minimum value of $f(x) = x$ for $x \ge 0$. Differentiation is of no help in finding the solution at $x = 0$.

Without loss of generality assume $b > a$ so $a + 100 = b$.

The product is $ab = a(a+100) = a^2 + 100a$.

To find an extrema we must solve for $f(x) = x^2 + 100x$ $f'(x) = 0$.

That is $2x + 100 = 0$ so $x = -50$ is the only extremum.

To see what type of extremum it is we must evaluate $f''(x)$ is at $x = -50$. $f''(x) = 2$ so $f''(50) = 2 > 0$. $x = 50$ it is a minimum.

So $b = 50$ and $a = -50$ is the minimum product. But those aren't positive.

We need to find the minimum positive product. If $0 < x$ then $f'(x) > 2*0 + 100$ so the product is increasing.

So the minimum product occurs at $a = \min (0, \infty)$ and $b = \min (100, \infty)$.

But .... there are no such real numbers.

Which... isn't a problem. You fell for the oldest trick in the book-- one which every mathematician I have ever known has fallen for once or twice--- that just because a problem might ask for something, that doesn't mean the thing exists.

Now had it be two non-negative real numbers the question would have been valid, but it wasn't and it isn't.

The minimum does not exists. There exists a minimizing sequence: $$ (a_n,b_n)=(1/n,100+1/n) $$

If you are trying to find an optimum, you need first to verify that an optimum exists. In this instance, one does not: as $b$ gets closer and closer to $0$ from above, $b(100+b)$ gets closer and closer to $0$ from above.