Demystify integration of $\int \frac{1}{x} \mathrm dx$

If you want to try to prove $\int\frac{\mathrm dx}x=\ln x + C $ (for $x \gt 0$), try the substitution

$$ \begin{align} x &= e^u \\ \mathrm dx &= e^u \mathrm du \end{align} $$

This substitution is justified because the exponential function is bijective from $\mathbb{R}$ to $(0,\infty)$ (hence for every $x$ there exists a $u$) and continuously differentiable (which allows an integration by substitution).

$$\int\frac{\mathrm dx}x=\int\frac{e^u\mathrm du}{e^u}=u+C$$

Now just use the fact that natural log is the inverse of the exponential function. If $x=e^u,u=\ln x$.

Let's use your result : $$\int x^a \, dx = \frac{x^{a+1}}{a+1} + C=\frac{x^{a+1}-1}{a+1} +C'$$ and take the limit as $\,a\to -1\,$ (since $\;x^{a+1}=e^{(a+1)\ln(x)}$) : $$\lim_{a\to\,-1}\int x^a \, dx = C'+\lim_{a\to\,-1}\frac{e^{(a+1)\ln(x)}-1}{a+1}=C'+\ln(x)$$ Hoping this will help your intuition,

Logarithms are all about solving exponential equations right? For example, logarithms are useful if you want to solve $2^x=3$. Maybe you are really confused about how in the world finding the area under $y=\frac{1}{x}$ has anything to do with solving such equations. Hopefully this post will illuminate that - but it may be a bit long winded.

Say you wanted to solve $2^x=3$ and you are a 16th century mathematician. So, no calculator. You could just use bisection method, getting closer and closer approximations, but this involves the very laborious process of computing roots, and you wouldn't want to have to do this every time you have an exponential equation to solve. So here is the genius idea. You will pay someone to calculate $1.0001^n$ for you for $n=1$ to $1000000$. You can teach them the mechanical skill of multiplying these numbers in a week because you only have to shift and add (see what I mean by computing the first few!). Then you look in your table to find that $1.0001^{6931} \approx 2$ and $1.0001^{10986} \approx 3$.

This helps you because now $2^n = 3$ can be rewritten as $1.0001^{6931n} = 1.0001^{10986}$, so $n = 10986/6931 \approx 1.585$. Just to double check ourselves, $2^{1.585} \approx 3.00007798$, so it pretty much worked.

This same table allows you to approximate the solutions to a huge number of exponential equations, in exactly the same manner. The upfront investment of labor is worth the time, because now you and your buddies can all solve these equations just by table lookups.

Later on, you want more precision, so you pay a whole group of laborers to calculate the powers of $1.000001$. You notice that in the old table for $1.0001$, the entry for $1.0001^{10000} = 2.71814593$, and the entry for $1.000001^{1000000} = 2.71828047$. Seems like a pretty special number to be coming up in these exponential tables, so you call it $e$ for exponential.

You have done all you can with exponential equations for the time being, so you turn your attention to other matters. First you notice the curious fact that the number of faces plus the number of vertexes minus the number of edges of any polyhedron seems to always be 2. You struggle to find a proof of this, but are tormented to no end. Later you turn to predicting the motion of the planets. Some time later your patron wants you to design some kind of war machine. Finally, many years after your experiments with exponential equations, you turn your attention to the problem of finding areas bounded by hyperbolas. You know that there are classical solutions to the area under a parabola, but as far as you know, no one has tackled the area under $y = \frac{1}{x}$ from $x=1$ to $x=2$, for instance.

You set about attempting to calculate this area. Your first thought is just to carve the region up into a great number of rectangles, all of equal base. You choose cut up the interval $[1,2]$ at the points $1.0001, 1.0002, 1.0003, 1.0004, 1.0005 ,... 1.9999$ and sum the areas of all the rectangles you get. You find that the answer is about $.693122$. Gifted as you are with the supernatural recall of a 16th century mathematician, you remember that all those years ago, $1.0001^{6931} \approx 2$. What a great mystery! What could be at the bottom of this coincidence?

First of all, in your old tables you were repeatedly multiplying 1.001, which is not exactly how you cut up [1,2] in your area calculation. However, there is nothing wrong with cutting up the interval [1,2] at the points $1.001, 1.001^2, 1.001^3, 1.001^4,$ etc. These numbers are all still close to each other, so computing the area of the corresponding rectangles will give a good approximation to the area under your curve, and these rectangles at least have a chance at relating the two problems.

So what are the areas of these rectangles? The area of the first rectangle is $1 \cdot (1.001-1) = .001$. The area of the second rectangle is $\frac{1}{1.001} \cdot (1.001^2-1.001) = .001$. The area of the third rectangle is $\frac{1}{1.001^2} \cdot (1.001^3-1.001^2) = .001$! Hey! These are all the same area! So the total area under the curve is just going to be $A = .001n$, where n is the solution to $1.001^n=2$. This is only an approximation though. We would get a better approximation if we used our table for 1.000001. It would be handy to write our result in a way which was agnostic about which table we were using. Oh ya, that number e always appeared in our table. So, we could say

$1.001^n = 2 \implies (1.001^{1000})^{0.001n} = 2 \implies e^A = 2$

Wow! Carrying out the same reasoning for any other $t$, we find that the area $A_t$ from $x=1$ to $x=t$ under $\frac{1}{x}$ should satisfy $e^{A_t} = t$. In other words this area function is giving an inverse to the exponential function with base e.

Hopefully this little story helps you understand what finding $\displaystyle\int_1^t \frac{1}{x} dx$ has to do with solving exponential equations, and also why the number $e$ comes into the picture naturally. To summarize, you cut up the interval [1,t] at the points $1, 1.001^2, 1.001^3, ..., 1.001^n$, where $n$ is the largest power of $1.001$ which is still less than $t$. To get from one rectangle to the next, the width gets multiplied by 1.001 and the height gets divided by 1.001, so the areas of all of these rectangles are the same. Thus the area under the rectangle $A$ is just $.001n$. But then $(1.001^{1000})^{.001n} \approx t$, and so $e^{.001n} \approx t$, which means $e^{A} \approx t$. This approximation gets better and better if you choose $1.000001$, or $1.00000000001$. So we actually have the equality $e^A =t$. Using the definition of natural logarithm as inverse of exponential function, we have $A = \ln(t)$.

I would recommend playing with this by writing a computer program to solve $a^n=b$ using the method we discussed (limit yourself to only addition, multiplication, and for loops). If you do not know how to program, just making a spreadsheet would be pretty informative: you can have your computer take the place of your laborer. Draw pictures of the area under the curve, partitioned in the way we have discussed, and see how that ties in. There is a lot to digest here.

The rule doesn't work when $a = -1$. There is a nice way you can get at the derivative of an inverse function if you know the derivative of the function, by way of something called implicit differentiation. Thus if you are willing to grant that $\frac{d}{dx} e^x = e^x$, then I can show that $\frac{d}{dx} \ln x = 1/x$, which is the result that you want. Write $y = \ln x$, then $e^y = x$ so that by the chain rule, $$\frac{d}{dx} e^y = e^y\frac{dy}{dx}$$ thus $$1 = e^y \frac{dy}{dx}$$ thus $$1 = x \frac{dy}{dx}$$ thus $$\frac{dy}{dx} = 1/x$$ which by definition is equivalent to the equation $$ \int \frac{1}{x} dx = \ln x +c $$