Indefinite integral question: $\int \frac{1}{x\sqrt{x^2+x}}dx$

Here is an approach.

$$ \begin{align} \int \frac{1}{x \sqrt{x^2+x}}dx &=\int \frac{1}{\sqrt{1+\frac{1}{x}}}\frac{dx}{x^2}\\\\ &=-\int \frac{1}{\sqrt{1+u}}\:du \quad (u=1/x)\\\\ &=-2\sqrt{u+1}+C\\\\ &=-2\sqrt{\frac{x+1}{x}}+C \end{align} $$ where $C$ is any constant.

If you don't want to have to "spot" which substitution to use, but instead want a "cookbook" approach, then consider using an Euler substitution. This would work here because you have a rational function of $x$ and $\sqrt{ax^2 + bx + c}$, in particular with $a=1$, $b=1$, and $c=0$.

You can use Euler's first substitution if $a>0$: here we write $\sqrt{ax^2+bx+c} = \pm x\sqrt{a}+t$ which gives $x = \frac{c-t^2}{\pm 2t\sqrt{a}-b}$. Since $a=1$ that's an option, but neither an $x+t$ nor $-x+t$ in the denominator look particularly helpful here because of the multiplication by $x$. If that had been an addition or subtraction of $x$ instead, then this substitution would have been ideal as that would have cancelled.

According to Wikipedia, we can use Euler's second substitution if $c>0$: here we write $\sqrt{ax^2+bx+c} = xt\pm\sqrt{c}$ which gives $x = \frac{\pm 2t\sqrt{c}-b}{a-t^2}$. Since $c=0$ I will not pursue this.

Euler's third substitution is used if $ax^2+bx+c$ has real roots $\alpha$ and $\beta$; we write $\sqrt{ax^2 + bx + c} = \sqrt{a(x-\alpha)(x-\beta)} = (x-\alpha)t$ which gives $x = \frac{a\beta-\alpha t^2}{a-t^2}$. This multiplicative form seems more suited for our purposes. In our case $x^2 + x = x(x+1)$ so we can take $\alpha=0$ and $\beta=-1$.

Pursuing the third substitution we obtain immediately $\sqrt{x^2 + x} = \sqrt{x(x+1)} = xt$ whence $t=\frac{\sqrt{x(x+1)}}{x} = \sqrt{\frac{x+1}{x}}$ and $x=\frac{1(-1) - 0t^2}{1-t^2}=(t^2-1)^{-1}$. Then $\frac{\mathrm{d}x}{\mathrm{d}t}=-2t(t^2-1)^{-2}$. Putting this together,

$$\int \frac{\mathrm{d}x}{x\sqrt{x(x+1)}}= \int \frac{-2t(t^2-1)^{-2}\mathrm{d}t}{x^2t} = \int \frac{-2t(t^2-1)^{-2}\mathrm{d}t}{(t^2-1)^{-2} t} = \int -2 \mathrm{d}t = -2t + C$$

Expressing back in terms of $x$ gives $-2 \sqrt{\frac{x+1}{x}} + C$. The advantage of this approach is that no "cleverness" is needed to find the required substitution, we just work through a checklist and see which one applies. A cleverer solution might be faster to perform, but take longer to spot.

Going back to Euler's second substitution, we can see that even though $c=0$ this would have worked just fine, and in fact it would have immediately given the same result as Euler's third substitution. I have the feeling the "$>$" in the Wikipedia article should really say "$\geq$" - but it doesn't make difference, since when $c=0$ the quadratic would factorise with one of the roots as zero and we can apply the third substitution as above.

Finally, reconsidering the first substitution, we could have gone ahead and put $\sqrt{x^2 + x} = x + t$ and $x = \frac{-t^2}{2t-1} = t^2 (1-2t)^{-1}$. By product rule,

$$\frac{\mathrm{d}x}{\mathrm{d}t}=2t(1-2t)^{-1} + 2t^2(1-2t)^{-2} = 2t(1-t)(1-2t)^{-2}$$

Substituting into the integral and clearing up is rather messy, for the reasons mentioned above, but can be done. Skipping quite a lot of algebra,

$$\int \frac{\mathrm{d}x}{x\sqrt{x(x+1)}}= \int \frac{2t(1-t)(1-2t)^{-2}\mathrm{d}t}{t^2 (1-2t)^{-1} ( t^2 (1-2t)^{-1} + t)} = \int 2t^{-2} \mathrm{d}t = -2t^{-1} + C$$

In terms of $x$, this is $\frac{2}{x - \sqrt{x^2 + x}} + C$. This is an alternative form to that given by Euler's third substitution, but was harder work to acquire. So don't just jump straight for the first substitution you see that fulfils the necessary conditions on the coefficients; spend a moment considering which of the available options best fits the form of the question.