Approximation of Riemann integrable function with a continuous function

Since $f$ is Riemann integrable, it is bounded, say $|f| \le M$. Now, given $\epsilon>0$ we can find a continuous $g$ such that $|g| \le M$ and

$$ \int_a^b |f-g| \, dx < \frac{\epsilon}{2M}$$

For example find a continuous $h$ which approximates $f$, i.e. $\int |f-h| \, dx < \epsilon/2M$ and set $$g(x) = \begin{cases} \min(M, h(x)) & h(x) \ge 0 \\\ \max(-M, h(x)) & h(x) < 0\end{cases}$$

Then, with this $g$, we have

$$\int_a^b |f-g|^2 \, dx \le \int_a^b |f-g|(|f|+|g|)\, dx \le 2M \int_a^b |f-g| \, dx < \epsilon$$

Note: Instead of using Riemann integrable in the usual sense here, I have used it in the sense of convergent improper integrals. If the discussion seems more complicated than is necessary, that is why.

You need to add in the condition that $f^2(x)$ is Riemann integrable. To see why, consider the function $f(x)=x^{-3/4}$ if $x\in (0,1]$ and $f(0)=0$. This is Riemann integrable, but for any continuous function on $[0,1]$ $(f(x)-g(x))^2$ will NOT be integrable. So let us add this condition to our discussion.

If you replace "Riemann integral" replaced with "Lebesgue integral", this result is standard material in any course on measure theory. More generally, one can show that continuous functions are dense in $L^p([a,b])$, the collection of functions that are integrable when you take them to the $p$th power. In your particular case, the integral you are using is the (square of) the distance in $L^2([a,b])$ between $f$ and $h$. Because Riemann integrable functions are also Lebesgue integrable functions, the result follows.

I do not believe that you can prove the second result strictly from the first because, even if $\int_a^b |f(x)-g(x)|dx$ is small, there could be places where $|f(x)-g(x)|$ is very large, and so integrating the difference squared might be infinite. You can construct an example by using the fact that $\sum \frac{n^2}{n^4}<\infty$ but $\sum \frac{(n^2)^2}{n^4}$ diverges. Since good approximations without squaring are not necessarily good approximations with squaring, you need to actually be precise about how you are doing your approximating.

If you add the hypothesis that $f$ is bounded, the new result will follow from the old one, because you can bound $\int_a^b |f(x)-g(x)|^2dx<(\max |f-g|)\int_a^b |f(x)-g(x)|dx$

If you want to see how everything is done in the case of the Lebesgue integral, I recommend Rudin's "Real and Complex Analysis". (I would include a link to google books, but the book is not readable there).