$a$ point is chosen randomly in $[0,5]$, $b$ is chosen randomly from $[1,6]$. Find $P(\int_\sqrt{b}^a xdx>\frac{3}{2})$?

As $a$ and $b$ are randomly chosen and considering independently chosen, $A \sim U(0, 5)$ and $B \sim U(1, 6)$.

Given $a^2 - b \gt 3$, yes we must have $a \gt 2$. But it is not correct that $a \lt 3$. For $2 \lt a \lt 3, 1 \lt b \lt a^2 - 3$ and for $a \gt 3$, any value of $b \in (1, 6)$ satisfies the condition.

So, $~ \displaystyle P(a^2 - b \gt 3) = \int_2^3 \int_1^{a^2-3} \frac{1}{25} ~db ~ da ~ +$ $ \displaystyle \int_3^5 \int_1^6 \frac{1}{25} ~db ~ da ~$

Or it is easier to see that for any value of $b$, we must have $a \gt \sqrt{b+3}$

So $\displaystyle P(a^2 - b \gt 3) = \int_1^6 \int_{\sqrt{b+3}}^5 \frac{1}{25} ~ da ~ db$

See if the diagram showing support of $a$ and $b$ helps. To the left of the parabola curve, $a^2 - b \lt 3$ and to its right, $a^2 - b \gt 3$. We need to find the probability that $a$ and $b$ are chosen in the region shaded in dark grey, which is to the right of parabola.