Integration using "method of judicious guessing"

Of course one can say: The expertise of the author led him to this "Ansatz". Note that you cannot make a mistake by trying it. If the true solution is not in the bag of proposed functions the computation will reveal this.

Your own efforts resulted in the recursion formula $$\int x^m\log^n x\>dx=\langle{1\over m+1}x^{m+1}\log^n x\rangle-{n\over m+1}\int x^m \log^{n-1} x\>dx\qquad(n\geq1)\ .$$ Here $m$ is fixed, and $n$ is the recursion variable, and the $\langle\ldots\rangle$ notation means "up to an additive constant". It is then pretty obvious, and can be verified by a simple induction proof, that the complete solution will be of the form $$\int x^m\log^n x\>dx=\bigl\langle x^{m+1}\sum_{k=0}^n c_k\log^{n-k}x\bigr\rangle\tag{1}$$ with certain rational coefficients $c_k$, e.g., $c_0={1\over m+1}$. Knowing this it is perhaps simpler to start right away with the "Ansatz" $(1)$ with indetermined coefficients $c_k$ than going through an $n$ steps of the recursion with all its integral signs.

But there is also an " abstract nonsense" proof that the proposed approach works: The result shows that it is not sufficient to consider "monomials" of the form $x^m\log^n x$ and integrate them in a single step. For a general theory we have to deal with the full space of functions $f$ of the form $$f(x)=x^m p(\log x)\qquad(x>0)\ ,\tag{2}$$ where $p(t)$ is a polynomial in an "indeterminate" $t$.

Denote the vector space of all polynomials $p(t)$ of degree $\leq n$ by $P_n$, and denote the vector space of all functions $(2)$ with $p\in P_n$ by $V_{m,n}$. For an $f\in V_{m,n}$ one computes $$f'(x)=x^{m-1}\bigl(m p(\log x)+p'(\log x)\bigr)\ ,\tag{3}$$ which implies that $f'\in V_{m-1,\, n}\>$.

This shows that the derivation $D:\>f\mapsto f'$ maps $V_{m,n}$ to $V_{m-1,\, n}$. The problem at hand is about the converse direction: We are given a $g\in V_{m-1,\,n}$ and have to show that $g$ is the $G'$ of some $G\in V_{m,n}$, in other words: that $$D:\>V_{m,n}\to V_{m-1,\, n}\ ,\qquad f\mapsto f'\tag{4}$$ is surjective.

In $(3)$ we see a linear map $\psi:\>P_n\to P_n$ at work, namely $$\psi:\quad P_n\to P_n,\qquad p\mapsto mp +p'\ .$$ When $m>0$ this map is injective, because $\psi(p)=0$, i.e., $p'=-mp$, implies $p=0$, or $p$ would be an exponential function. As $P_n$ is finite dimensional we can conclude that $\psi$ is also surjective, hence $(4)$ is surjective as well. This is saying that any $g\in V_{m-1, \,n}$ is the derivative of some $G\in V_{m,\,n}$. In particular this is the case for the function $g(x):=x^{m-1}\,(\log x)^n$.

More or less, when the author of the textbook makes their guess of the form of the integral, they are eliding exactly the work that you did to calculate the integral.

Essentially, there is an inductive argument at work here. So, the critical observation that the textbook asks you to make is the following one: $$f_{m+1,n}'=m\cdot f_{m,n}+n\cdot f_{m,n-1}.$$ The idea is then that if you want to integrate $f_{m,n}$, what you need to do is the following:

• Calculate an anti-derivative for $f_{m,n-1}$.

• Subtract $n$ times that anti-derivative from $f_{m+1,n}$ and divide by $m$.

The idea then is that, if you repeat this argument until you get to calculating an anti-derivative for $f_{m,0}$ - which is just some multiple of $f_{m+1,0}$ - you see that the answer should be some weighted sum of the following functions $$f_{m+1,n},\,f_{m+1,n-1},\,f_{m+1,n-2},\ldots,f_{m+1,2},\,f_{m+1,1},\,f_{m+1,0}.$$ This is just what happens when you "unroll" the process to figure out the antiderivative of $f_{m,n}$ by using the antiderivative for $f_{m,n-1}$, which can be determined by knowing the antiderivative for $f_{m,n-2}$ and so on.

Basically, the textbook runs this argument by figuring out what the terms will be, ignoring the factors of $n$ and $m$ that appear, then figuring out what the coefficients have to be at the end. Your method does the same idea, but you explicitly keep track of the constants at each step.

The clever step here is in seeing that $f_{m+1,n}'=m\cdot f_{m,n} + n\cdot f_{m,n-1}$. Once you have that, you see that you can relate your desired anti-derivative to a simpler one, and then proceed. I would say that the "judicious guess" is this step. As you discovered, there are multiple ways to proceed once you get to this step - and whatever you do, it's a somewhat straightforward calculation from that point on.