The deep reason why $\int \frac{1}{x}\operatorname{d}x$ is a transcendental function ($\log$) [duplicate]

I'll try to give a soft answer to what I see as the spirit of the question, which is not why you get exactly log, but why the behaviour is different when integrating $x^{k}$ for $k=-1$.

The way I see it is that the logarithm would actually be there for other powers of $x$ too, but it's "hidden" by the fact that in a geometric series one term "gobbles up" all the others together, except in the very special case when all terms are equal. Put in other words, the intuition is very close to the reason why $\sum_{i=a}^b c^i$ can be reasonably approximated by just the first term of the sum ($c^a$) if $c<1$, and just by the last ($c^b$) if $c>1$, regardless of how large $b-a$ is, i.e. of how many terms you have in the sum. But if $c=1$ no single term dominates all the others, and that's when you have to count them all, and you end up seeing the $b-a$ term "emerge" in $\sum_{i=a}^b 1^i=b-a+1$.

Informally, you can see how this applies to the case at hand writing $\int_{x_0}^{x_f} x^k dx$ as $\int_{x_0}^{2 x_0} x^k dx + \int_{2x_0}^{4x_0} x^k dx + ...$. Each of the $\approx \log_2 \frac{x_f}{x_0}$ terms has the same weight as the others if, and only if, $k$ has a very specific value (which?), in which case $\int_{x_0}^{x_f} x^k dx$ equals $\approx \log_2 \frac{x_f}{x_0}$ times $\int_{x_0}^{2 x_0} x^k dx$. If it's just an $\epsilon$ smaller, or larger, you get the sum within a constant factor of either $\int_{x_0}^{2 x_0} x^k dx$ or of $\int_{\frac{1}{2}x_f}^{x_f} x^k dx$, respectively, regardless of $\frac{x_f}{x_0}$. Note that instead of using $2$ as a base, we could have used $3$, or $e$, or $7.24$, or $\pi$, and the critical value of $k$ would have remained the same: it's the value that ensures that if you integrate $x^k$ over an interval $7.24$ longer, but with a starting point $7.24$ times larger, the integral does not change, $k=-1$.

This is actually a phenomenon that I've seen pop up really really often in math, physics, and computer science. You often have a sum of many terms, and a parameter that, for small values, makes the first term of the sum dominate all the others put together, and for large values, makes the last term dominate all the others put together, regardless of how many terms you have in the sum. But when the parameter equals exactly the critical value at which the transition between the two regimes occurs, all sort of strange quantities related to the number of terms in the sum appear in any approximation of it.

The so-called "deep reason" is not deep at all. The term $\displaystyle \log(x)$ is simply the constant term in the expansion of $\displaystyle \frac{x^{n+1}}{n+1}$ around $n=-1$. To see this, we simply write $$\begin{align}\frac{x^{n+1}}{n+1}&=\frac{e^{(n+1)\log(x)}}{n+1}\\\\&=\frac{1}{n+1}\sum_{k=0}^\infty \frac{(n+1)^k\,\log^k(x)}{k!}\\\\&=\sum_{k=0}^\infty\frac{(n+1)^{k-1}\log^k(x)}{k!}\\\\&=\frac{1}{n+1}+\log(x)+\frac12(n+1)\log^2(x)+O((n+1)^2)\end{align}$$

whence we see the leading terms in the asymptotic ($n\sim -1$) expansion of $\displaystyle \frac{x^{n+1}}{n+1}$. Obviously, we see that $\lim_{n\to -1}\frac{x^{n+1}-1}{n+1}=\log(x)$.

More simply, let $f(x,n)$ be given by the integral

$$\begin{align} f(x,n)&=\int_1^x t^n\,dt\\\\ &=\frac{x^{n+1}-1}{n+1}\tag 1 \end{align}$$

Note that $f(x,n)$ is continuos on $(0,\infty)\times \mathbb{R}$ (i.e., $n$ need not be restricted to the integers).

Then, note that the limit as $n\to -1$ of $f(x,n)=f(x,-1)$ is

$$\begin{align} \int_1^x \frac1t\,dt&=f(x,-1)\\\\ &=\lim_{n\to -1}f(x,n)\\\\ &=\lim_{n\to -1} \frac{x^{n+1}-1}{n+1}\\\\ &=\lim_{n\to -1}\frac{e^{(n+1)\log(x)}-1}{n+1}\\\\ &=\log(x) \end{align}$$

So, we can recover the expected result, $\int_1^x \frac{1}{t}\,dt=\log(x)$, by taking the limit in $(1)$ for $n\ne -1$.