Unsure of my work evaluating $\int \frac{dx}{\sqrt{x + \sqrt{x + \sqrt{x + \cdots}}}}$
Solution 1:
Short Answer:
Your work is perfectly fine if your lower and upper integral limits satisfy $0 < a \leq b$. In that case your answer $2u - \ln(u)$ even has a nice closed form purely in terms of $x$:
$$ \int_a^b \frac{dx}{\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}} = \Big[\big(1 + \sqrt{4x + 1}\big) - \ln\Big(\frac{1}{2}\big(1 + \sqrt{4x + 1}\big)\Big)\Big]_a^b $$
This formula also continues to work for a lower limit of $a = 0$ if you interpret either the integral and/or the nested radical $\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}$ in the denominator properly enough.
Long Answer (Analysis):
Since you used $u$-substitution, your method should work as long as the conditions for an integration by substitution are met. Say you are integrating over some interval $[a, b]$. You have to verify:
Does the function $u(x) = \sqrt{x + u(x)}$ that you defined implicitly actually make sense over $[a, b]$? In other words, is there really a function $u : [a, b] \to \Bbb R$ that satisfies that recursion?
Is the function $u(x)$ actually differentiable over $[a, b]$?
$\underline{\textit{There is some good news for these questions:}}$
As long as $x > 0$, there is a well-defined expression for $u$ in terms of $x$ when $u = \sqrt{x + u}$. To realize this, we need translate the intuitive expression $\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}$ into the precise language of calculus. Only then can we bring the full power of calculus to bear on this problem. So formally what is going on with a nested radical like $\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}$ is this:
Let $u_{x,1} = \sqrt{x}$ and define recursively the sequence $u_{x,n + 1} = \sqrt{x + u_{x,n}}$ ($n \in \Bbb Z_+$). If $$u_x = \lim\limits_{n \to \infty}u_{x,n}$$ exists, then we may define our sought-after function $u$ at $x$ to be $u(x) = u_x$. In essence, the limit $\lim\limits_{n \to \infty}u_{x,n}$ is mathematically what we define the expression $\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}$ to be. And we can easily check that $u_x = \sqrt{x + u_x}$ by taking the limit as $n \to \infty$ at both sides of the equation $u_{x,n + 1} = \sqrt{x + u_{x,n}}$.
Now the good news is that, as long as $x > 0$, you can show that the sequence $u_{x,n}$ is bounded and monotonically increasing so that it does converge to a definite limit, namely
$$u_x = \frac{1}{2}\big(1 + \sqrt{4x + 1}\big)$$
Hence, our function $u(x) = u_x$ is well-defined for $x > 0$. Also, note that the formula above should not surprise you. You can easily see where it originated:
Informally, if you take your substitution equation $u^2 - u = x$ and wrote it as a quadratic equation $u^2 - u - x = 0$, you can solve it by thinking of $x$ as a constant. And indeed, one of the solutions that pops out is precisely $u_+ = \frac{1}{2}\big(1 + \sqrt{4x + 1}\big)$. You can eliminate the other solution $u_- = \frac{1}{2}\big(1 - \sqrt{4x + 1}\big)$ since it is negative if $x > 0$ and by convention square roots are positive.
So as long as $x > 0$, you can safely take $$u(x) = \frac{1}{2}\big(1 + \sqrt{4x + 1}\big)$$ as the $u$-substitution function which satisfies $u = \sqrt{x + u}$. In fact, as is apparent from the formula, $u(x)$ is even differentiable in this case.
$\underline{\textit{But there are caveats:}}$
$1.\ \textbf{Note that for $x < 0$, the limit does not make sense:}$ as the very first sequence element $u_{x,1} = \sqrt{x}$ is not real. So, from this very analysis, you can immediately conclude that you should not be integrating over negative values in your integral.
$2.\ \textbf{Next, at $x = 0$, things almost work out but break down anyway:}$ Note, we only managed to eliminate $\frac{1}{2}\big(1 - \sqrt{4x + 1}\big)$ as a candidate for the limit above because it was negative if $x > 0$. Well, if $x = 0$, then $u_- = \frac{1}{2}\big(1 - \sqrt{4x + 1}\big) = 0$ and you can no longer eliminate it that easily. So, we must go back to our definition of $\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}$ in terms of sequences to arbitrate between $u_+$ and $u_-$. Applying that definition when $x = 0$, we see that $u_- = 0$ is the candidate that is chosen this time not $u_+ = 1$. This is because in this case, all the sequence elements $u_{x,n}$ are zero: $$u_{x=0,1} = \sqrt{x} = \sqrt{0} = 0,\quad u_{x=0,2} = \sqrt{x + u_{x=0,1}} = \sqrt{0 + 0} = 0,\quad \ldots \text{ etc}$$
Hence, $\lim\limits_{n \to \infty}u_{x=0,n} = 0$ and $u(0) = 0$. However, approaching $0$ from the right, we see that $$\lim\limits_{x \to 0+}u(x) = \frac{1}{2}\big(1 + \sqrt{4\cdot0 + 1}\big) = 1$$ And therefore even though $u(x)$ is defined at $x = 0$, it is sadly not continuous there, let alone differentiable. So the $u$-substitution Theorem no longer applies.
In any case, there is an even worse problem when $x = 0$. Note that the function you are trying to integrate $f(x) = \sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}$ is undefined at $x = 0$ because as we saw, our definition of $\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}$ in terms of sequences gives you a $0$ when $x = 0$. So there would be a $0$ in your denominator for $f(x)$ if that was allowed.
$\underline{\textit{Okay, so we have concluded so far that:}}$
As long as your integration interval $[a, b]$ satisfies $0 < a \leq b$, your work should go through and you can use $u(x) = \frac{1}{2}\big(1 + \sqrt{4x + 1}\big)$ as the explicit formula for $u$ to express your final integral answer:
$$ \int_a^b \frac{dx}{\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}} = \Big[\big(1 + \sqrt{4x + 1}\big) - \ln\Big(\frac{1}{2}\big(1 + \sqrt{4x + 1}\big)\Big)\Big]_a^b $$
$\underline{\textit{Fixing the breakdown at $x = 0$:}}$
If you really want $x = 0$ as one of the limits e.g. $$ \int_0^b \frac{dx}{\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}} $$ for $b > 0$, you can do so in two ways, both of which lead to the same result:
You can modify the definition of $\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}$ thus: it defaults to the usual definition via sequences if $x > 0$ and to $\frac{1}{2}(1 + \sqrt{4 \cdot 0 + 1}) = 1$ if $x = 0$. Then you can safely use $u(x) = \frac{1}{2}\big(1 + \sqrt{4x + 1}\big)$ for all $x \geq 0$. And the answer you will get for your integral is exactly what you would expect by plugging in $a = 0$ in the closed form I gave above: $$ \big(1 + \sqrt{4b + 1}\big) - \ln\Big(\frac{1}{2}\big(1 + \sqrt{4b + 1}\big)\Big) - 2 $$
On the other hand, you can instead take a limiting integral in the same spirit that we define $\int_0^b \frac{1}{x^2}dx$ to get around the singularity of $\frac{1}{x^2}$ at $0$. That is, you can define: \begin{align*} \int_0^b \frac{dx}{\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}} &:= \lim_{a \to 0+}\int_a^b \frac{dx}{\sqrt{x+\sqrt{x+\sqrt{x+\ldots}}}} \\ &= \lim_{a \to 0+}\big[2u(x) - \ln(u(x))\big]_a^b \end{align*} This leads to the same answer because ultimately $\lim\limits_{a \to 0^+}u(x) = 1$.