Proof of analogue of the Cauchy-Schwarz inequality for random variables

Look at the following two expectations:

$$E[(aX+bY)^2]=a^2E[X^2] + b^2E[Y^2] + 2abE[XY] \ge 0$$ $$E[(aX-bY)^2]=a^2E[X^2] + b^2E[Y^2] - 2abE[XY] \ge 0$$

Now let $a^2=E[Y^2]$ and $b^2=E[X^2]$. This gives

$$ 2abE[XY] \ge -2a^2b^2$$ $$ 2abE[XY] \le 2a^2b^2$$

Dividing by $2ab$ results in

$$ -\sqrt{E[X^2]} \sqrt{E[Y^2]} \le E[XY] \le \sqrt{E[X^2]} \sqrt{E[Y^2]}$$ which is equivalent to

$$ |E[XY]| \le \sqrt{E[X^2]} \sqrt{E[Y^2]}$$

There is some debate about whether or not the mapping $(X, Y) \mapsto E[XY]$ defines an inner product. If it does, then your result follows from the standard Cauchy-Schwarz inequality in general inner product spaces.

But, even if it's correct, this approach is rather a cop-out, and could easily be regarded as circular reasoning.

To avoid the circular reasoning, just adapt any proof of the Cauchy-Schwarz inequality. A few of them are given on this wikipedia page.

Here is one such proof: As your hint says, we know that $E[(kX+Y)^2] \ge 0$, for any real $k$. Expanding this, we get:

$$k^2E[X^2] + 2kE[XY] + E[Y^2] \ge 0$$

Let's regard this as a quadratic in $k$. It is non-negative for all $k$, so it has at most one real root, so its discriminant (the $b^2 - 4ac$ thing) must be $\le 0$. In other words $$ (2E[XY])^2 - 4E[X^2]E[Y^2] \le 0$$ and the result follows immediately.

Note that I didn't prove the statement about when equality occurs. In fact, I don't think this statement is true, as WimC's comment pointed out.