How to know which test to use on improper integral?

Solution 1:

Is this the correct method for solving this?

No. This is a correct method for solving the problem.

What is the general way of determining whether you should use direct comparison vs limit comparison for finding if improper integrals are convergent or divergent?

There isn't one. The only hard and fast rule is that direct comparison only works if you can show that the integrand is single-signed close to the improper limit. Otherwise, your comparison integral can converge, but the value of the original integral can oscillate instead as you approach the improper limit. In that case, direct comparison breaks down and you must find some other approach.

But when the integrand is single-signed near the improper limit, it would be difficult to find an example that yields to the limit comparison test that does not also yield to the direct comparison test, or vice versa. The comparison functions will not always be the same between the two, but the existence of a comparison function that works for one generally implies the existence of a comparison function that works for the other. Though the other comparison function may be difficult to find.

In your example case, the same comparison function works for both. $\tan^{-1} x <\frac \pi 2$ and $x^3 + \sqrt x > x^3$ on $[1,\infty)$, so $$0 < \frac{\tan^{-1} x}{x^3 + \sqrt x} < \dfrac{\frac\pi 2}{x^3}$$ Thus the direct comparison test also show the integral converges.

I normally look at the solutions and I'm able to understand what they are doing but I don't understand the thought process of choosing a specific test.

That one is easy: play around until you understand this particular problem enough to find a method that works. This is the general process for solving any math problem, and that is all there is to it.

There is no algorithm that says "do this, then this then this, and out pops the correct answer". You can find small classes of problems where you can write such an algorithm, but there is no human-performable algorithm that would handle all improper integral problems, not even all improper integral problems that are solvable by either the limit or direct comparison tests.

Furthermore, especially for Limit Comparison test, I often don't know how they choose g(x)

That is where "play around with it until you understand this particular problem enough" comes in. In this case, it is not hard to see how they came up with this comparison function: $\lim_{x \to \infty} \tan^{-1}x = \frac \pi2$ and, as $x\to \infty$ (or as $x\to 0$), higher powers of $x$ outpace lower powers of $x$. So when $x$ is large, $x^3$ dominates $\sqrt x$, whose contribution becomes tiny in comparison (consider the difference between the two when $x$ is one million, or one quadrillion).

So $$\tan^{-1}x \sim \frac \pi2\\x^3 + \sqrt x \sim x^3$$ and because division is continuous,$$\frac{\tan^{-1} x}{x^3 + \sqrt x} \sim \frac{\pi/2}{x^3}$$