Finding Moment of Inertia (Rotational Inertia?) $I$ Using Integration?

Seeing an expression like $I = \int r^2 dm$ is certainly confusing the first time you see it. When you see an expression like $I = \int x^2 dx$ we effectively iterating over a range on the x axis and adding up the area of an infinite amount of infinitesimally small strips. Note that the area of each strip is approximated as the function value (x^2) times the strip length (dx). When you see an expression like dm, we are iterating over masses instead. We still sum up the function value (r^2), but this time we multiply it by the strip mass (dm). To solve the problem, we usually put m in terms of another variable which we can iterate over more easily.

For example, consider the moment of inertia of a rod of length L around its center with total mass of L. Each bit of length (dx) has mass (dm) and r=|x|. Solving for $I = \int r^2 dm = \int |x|^2 dx = \int x^2 dx = (x^3)/3+c$. Now, we have definite values for x to sub in (-L/2 and L/2), so we write $$ I = \frac{(L/8)}{3}-\frac{(-L/8)}{3}=\frac{L}{12} $$

Now lets calculate the moment of inertia of the hoop case you described. We break the hoop up into infinitesimally small rings the same distance from the center. Let the hoop have inner thickness r and outer thickness R. The area is $R^2-r^2$. The area density, d, is therefore $\frac{m}{R^2-r^2}$. A ring at radius k with thickness dk has area $2\pi~k~\mathrm{d}k$, mass is $2\pi~k~\mathrm{d}k\frac{m}{R^2-r^2}$ and moment of inertia around the central axis $2\pi~k^3~\mathrm{d}k\frac{m}{R^2-r^2}$. The integral is $\frac{\pi}{2}k^4\frac{m}{R^2-r^2}+c$. Subbing in the exact values, we get $$ \frac{\pi}{2}(R^4-r^4)\frac{m}{R^2-r^2} = \frac{\pi}{2}(R^2+r^2)m $$

$dm$ takes density fluctuations into account, it's just $dm = \rho(\vec r) d^3 \mathbf r$.

E.g. for a homogeneous cylinder with the rotational axis align parallel to the z-axis it is $\rho = \frac{m}{V}\cdot\theta(R-r)\theta(a^2-z^2)$, where $\theta(x) = \begin{cases} 0 \;\text{ for } x<0 \text{ and}\\ 1 \;\text{ for } x>0 \end{cases}$ and $r = \sqrt{x^2+y^2}$ is the radial coordinate in cylindrical coordinates. The result is that your integration over $r$ is from $0$ to $R$ and for $z$ from $-a$ to $a$. $V=\pi R^2 \cdot 2a$ is the volume of the cylinder, $m$ its weight. In cylindrical coordinates, $d^3\mathbf r = r\, dr \,d\phi\, dz$, so you got

$$I = \int_{r=0}^{R} \int_{z=-a}^a \int_{\phi=0}^{2\pi} r^2 \frac{m}{V} \cdot r\, d\phi\, dz\, dr = 4a\pi \frac{m}{V} \int_{r=0}^R r^3 dr = \frac{am\pi}{V}R^4 = \frac{1}{2}mR^2.$$