Riemann-integrable (improperly) but not Lebesgue-integrable

First substitute $u= x^{\alpha}$ into the integral to see your problem reduces to asking the same question about the integral $\displaystyle \int^{\infty}_0 \dfrac{\sin x}{x^a} dx$ for $a\in (0,1).$

To show that is Riemann integrable, we consider the integral over $(0,1)$ and $[1,\infty)$ separately. To show the integral over $[1,\infty)$ exists, integrate by parts. You'll have

$$ \int^R_1 \frac{\sin x}{x^a} dx = -x^{-a} \cos x \mid^R_1 - a \int^R_1 \frac{\cos x}{x^{a+1}} dx$$

and it should be easy to finish from there.

To see $\displaystyle \int^1_0 \frac{\sin x}{x^a} dx$ exists note that $\sin x \sim x$ and $\displaystyle \int^1_0 \frac{x}{x^a} dx$ is finite.

To show it is not Lebesgue integrable, it suffices to show $$\int^{\infty}_0 \frac{|\sin x|}{x^a} dx = \sum_{k=0}^{\infty} (-1)^k \int^{(k+1)\pi}_{k\pi} \frac{\sin x}{x^a} dx = \sum_{k=0} \int^{\pi}_0 \frac{\sin x}{(x+k\pi)^a} dx$$ diverges, which is not hard with a basic estimate on the last integral. Note that the integral over $[0,\pi]$ is certainly greater than the integral over $[\pi/4,3\pi/4]$ and on that interval we have $\sin x \geq \frac{1}{\sqrt{2}}$ so

$$\int^{\pi}_0 \frac{\sin x}{(x+k\pi)^a} dx > \frac{1}{\sqrt{2}} \int^{3\pi/4}_{\pi/4} \frac{1}{( x+k\pi)^a} dx > \frac{1}{\sqrt{2}} \cdot \frac{\pi}{2} \frac{1}{( 3\pi/4 + k\pi)^a}= \frac{1}{2\sqrt{2}\pi^{a-1}}\frac{1}{(k+3/4)^a} .$$

I felt the counterexamples mentioned before are complicated and not straightforward to justify/remember. Instead, we can consider the following relatively simpler counterexample.

Consider the function $f(x)=\chi_{[n,n+1]}(x)\left(\frac{(-1)^n}{n}\right).$ $f$ is improper Riemann integrable and $\int\limits_{\mathbb{R}}f(x)dx=\sum\limits_{n \in \mathbb{N}}\frac{(-1)^n}{n} < \infty,$ but not Lebesgue integrable as $\int\limits_{\mathbb{R}} \left|f(x)\right|dx=\sum\limits_{n \in \mathbb{N}}\frac{1}{n}=\infty.$

For the first one, reduce to show that $\int_1^\infty\sin(t^\alpha)dt$ is convergent. A substitution like $s=t^\alpha$ and an integration by part will give what we want.

For the second one, use again the same trick, then the inequality $\sin^2s\leqslant|\sin s|$. A trigonometric identity will do the job.