How can I show that $f$ must be zero if $\int fg$ is always zero?

Solution 1:

Hint: set $g=f$. Then $\int_a^b f^2 \, dx =0$. Can you see why this implies $f=0$?

Solution 2:

You can work by assuming, that $f$ is not identically zero. Just for simpliciy, assume that $[a,b]=[-1,1]$ and $f(0)=f_0>0$. Since $f$ is continous, there exists $\epsilon>0$ such that $f(x)>f_{min}>0\quad \forall x\in [-\epsilon,\epsilon]$. Since your initial statement shall hold for all $g$, we choose $g = \chi_{-[\epsilon,\epsilon]}(x)$, i.e. $g(x)=1$ if and only if $x\in [-\epsilon,\epsilon]$. Then \begin{align} \int_a^bf(x)g(x)dx &= \int_{-1}^1 f(x) \chi_{-[\epsilon,\epsilon]}(x) dx\\ &=\int_{-\epsilon}^\epsilon f(x)dx >2\epsilon f_{min}>0 \end{align} which is a contradiction. Therefore, $f\equiv 0$.

Edit:

I didn't read, that $g$ has to be continous. But you can easily replace $g$ in my proof by a function that is for example a hat function or the bump function on $[-\epsilon,\epsilon]$. It is only important, to have a function that is zero everywhere except for the interval $[-\epsilon,\epsilon]$ so you can "pick" the specific region where you assume $f$ to be non-zero.