Maximum of $ F(f)=\int_0^1 |f(x)|^2\; dx-\left(\int_0^1 f(x)\; dx\right)^2 $ over a subset of continuous functions on $[0,1]$
Solution 1:
For the sake of convenience, we will consider the functional
$$ \mathsf{Var}(f) = \int_{0}^{1} f(x)^2 \, \mathrm{d}x - \left( \int_{0}^{1} f(x) \, \mathrm{d}x \right)^2 $$
on the set $\overline{X}$ of all measurable functions $f$ on $[0, 1]$ such that $0 \leq f(x) \leq x$ for all $x \in [0, 1]$. Note that $\overline{X}$ is the closure of OP's set $X$ in $L^2([0,1])$, justifying the notation.
In order to find the maximum of $\mathsf{Var}(\cdot)$, for each $c \in [0, 1]$ we define $g_c : [0, 1] \to \mathbb{R}$ by
$$ g_c(x) = x \mathbf{1}_{[c,1]}(x) = \begin{cases} x, & x \geq c, \\ 0, & x < c. \end{cases} $$
This function is related to our maximization problem via:
Lemma. Let $f \in \overline{X}$, and choose $c \in [0, 1]$ so that $\int_{c}^{1} x \, \mathrm{d}x = \int_{0}^{1} f(x) \, \mathrm{d}x$. Then $$ \mathsf{Var}(f) \leq \mathsf{Var}(g_c) $$ with equality if and only if $f = g_c$ almost everywhere.
Proof. Since $\int_{0}^{1} g_c(x) \, \mathrm{d}x = \int_{0}^{1} f(x) \, \mathrm{d}x$, we get
\begin{align*} \mathsf{Var}(g_c) - \mathsf{Var}(f) &= \int_{0}^{1} (g_c(x)^2 - f(x)^2) \, \mathrm{d}x \\ &= \int_{c}^{1} (x + f(x))(x - f(x)) \, \mathrm{d}x - \int_{0}^{c} f(x)^2 \, \mathrm{d}x \\ &\geq \int_{c}^{1} c(x - f(x)) \, \mathrm{d}x - \int_{0}^{c} c f(x) \, \mathrm{d}x \\ &= c \biggl( \int_{c}^{1} x \, \mathrm{d}x - \int_{0}^{1} f(x) \, \mathrm{d}x \biggr) \\ &= 0. \end{align*}
Here, we used the inequalities $x + f(x) \geq x \geq c$ for $x \geq c$ and $f(x) \leq x \leq c$ for $x \leq c$. Moreover, this tells that the equality holds if and only if $x - f(x) = 0$ for a.e. $x \in [c, 1]$ and $f(x) = 0$ for a.e. $x \in [0, c]$, proving the equality condition. $\square$
Using this, we are ready to answer OP's question. Let $\alpha = \frac{\sqrt{5} - 1}{2}$. Then it is easy to check that $c \mapsto \mathsf{Var}(g_c)$ is maximized at $c = \alpha$. Now, since $g_{\alpha}$ can be approximated in $L^2$ by the elements of $X$,
$$ \sup_{f \in X} \, \mathsf{Var}(f) = \mathsf{Var}(g_{\alpha}) = \frac{5}{24}(3 - \sqrt{5}) \approx 0.159153. $$
However, this supremum cannot be attained on $X$, because $f \in \overline{X}$ and $\mathsf{Var}(f) = \mathsf{Var}(g_{\alpha})$ together imply that $f = g_{\alpha} \notin X$.