Prove that:$f(f(x)) = x^2 \implies \int_{0}^{1}{(f(x))^2dx} \geq \frac{3}{13}$

It took a while, but I finally managed to have a solution.

I will show that $\displaystyle{\int_0^1{(f(x))^2dx} = 1-4\int_0^1{x^3f(x)dx}}$. (*)

For $n \in \mathbb{N}^*$, let $D_n = (\displaystyle{0, \frac{1}{n}, ..., \frac{n-1}{n}},1)$ and $E_n = (\displaystyle{0, f(\frac{1}{n}), ..., f(\frac{n-1}{n}),1})$ be two divisions on the interval $[0,1]$.

Evidently, $||D_n|| = \displaystyle{\frac{1}{n}}$, so $\displaystyle{\lim_{n \to \infty}{||D_n||} = 0}$.

Because $f$ is a continuous function, then it is also uniformly continuous, so $\displaystyle{\lim_{n \to \infty}{||E_n||} = 0}$.

Let $\xi_i = \displaystyle{\frac{i}{n}}$.

Now I will write the Riemann sum for $f^2$ for the division $E_n$(upper Darboux sum).

$S_{E_n}(f^2) = \displaystyle{\sum_{i=1}^n{(f(f(\xi_i)))^2(f(\xi_i) - (f(\xi_{i-1})))} = \sum_{i=1}^n{\xi_i^4(f(\xi_i)-f(\xi_{i-1}))}}$.

We can rewrite this sum:

$S_{E_n}(f^2) = \xi_n^4 f(\xi_n) - \xi_0^4 f(\xi_0) + \displaystyle{\sum_{i=1}^{n}{f(\xi_{i-1})(\xi_{i-1}^4 - \xi_i^4)} = 1- \sum_{i=1}^n{f(\xi_{i-1})(\xi_{i}^4 - \xi_{i-1}^4)} = 1 - \sum_{i=1}^n{f(\xi_{i-1})(\xi_i^3 + \xi_i^2\xi_{i-1} + \xi_i\xi_{i-1}^x + \xi_{i-1}^3)(\xi_i - \xi_{i-1})}}$.

Also, $4\xi_{i-1}^3 f(\xi_{i-1}) \leq f(\xi_{i-1})(\xi_i^3 + \xi_i^2\xi_{i-1} + \xi_i\xi_{i-1}^x + \xi_{i-1}^3)(\xi_i - \xi_{i-1}) \leq 4\xi_i^3 f(\xi_i)$.

Considering the function $h : [0,1] \to [0,1], h(x) = x^3f(x)$, we obtain that:

$s_{D_n}(h) < \displaystyle{\sum_{i=1}^n{f(\xi_{i-1})(\xi_i^3 + \xi_i^2\xi_{i-1} + \xi_i\xi_{i-1}^x + \xi_{i-1}^3)(\xi_i - \xi_{i-1})}} < S_{D_n}(h)$, where $s_{D_n}(h)$ and $S_{D_n}(h)$ represent the Darboux sums(lower and upper, respectively) for the function $h$ and the division $D_n$.

Therefore, we have that $\displaystyle{\lim_{n \to \infty}{\sum_{i=1}^n{f(\xi_{i-1})(\xi_i^3 + \xi_i^2\xi_{i-1} + \xi_i\xi_{i-1}^x + \xi_{i-1}^3)(\xi_i - \xi_{i-1})}} = \int_0^1{h(x)dx} = 4\int_0^1{x^3f(x)dx}}$.

Now, using the Cauchy-Schwarz inequality we have that:

$\displaystyle{\int_0^1{x^3f(x)dx} \leq \sqrt{\int_0^1{x^6dx} \cdot \int_0^1{(f(x))^2dx}} = \sqrt{\frac{\int_0^1{(f(x))^2dx}}{7}}}$. (**)

Let $a = \displaystyle{\int_0^1{(f(x))^2dx}}$.

Using (*) and (**), we obtain that $\displaystyle{0 < 1-a \leq 4\sqrt{\frac{a}{7}}}$, therefore $(1-a)^2 \leq \displaystyle{\frac{16a}{7}} \iff 7a^2 - 30a + 7 \leq 0 \implies a \geq \displaystyle{\frac{15 - \sqrt{176}}{7}}$.

Finally, $\displaystyle{\int_0^1{(f(x))^2dx} \geq \frac{15 - \sqrt{176}}{7} > 0,24 > \frac{3}{13}}$, so we are done.