Solution 1:

Let $C_1=\left[0,\frac{1}{3}\right]\cup\left[\frac{2}{3},1\right]$, $C_2=\left[0,\frac{1}{9}\right]\cup\left[\frac{2}{9},\frac{3}{9}\right]\cup\left[\frac{6}{9},\frac{7}{9}\right]\cup\left[\frac{8}{9},\frac{9}{9}\right]$ and so on the usual sets used to define the Cantor set. Then $\mu_F$ is the limit as $n\to +\infty$ of the probability measure $\mu_{P_n}$ on $C_n$. Let $I=[a,a+3b]$ be any closed interval of the real line and $J$ the same interval without its middle third, $J=[a,a+b]\cup[a+2b,a+3b]$. Then: $$ \int_I x^2 d\mu = \frac{1}{3}\left((a+3b)^3-a^3\right)=3b(a^2+3ab+3b^2), $$ $$\frac{3}{2}\int_J x^2 d\mu = 3b(a^2+3ab+3b^2)+b^3, $$ so: $$ \frac{3}{2}\int_J x^2 d\mu = \int_I x^2 d\mu + \frac{\mu(I)^3}{27},\tag{1}$$ giving immediately: $$ \int_{0}^{1} x^2\, d\mu_F = \lim_{n\to +\infty}\int_{0}^{1} x^2\, d\mu_{P_n} = \lim_{n\to +\infty}\sum_{k=0}^{n}\frac{1}{3^{2k+1}}=\color{red}{\frac{3}{8}} .\tag{2}$$

Solution 2:

Here is another calculation, based on how $\mu_F$ behaves under scaling $x \to \frac{1}{3} x$ and reflection $x \to 1-x$.

Let $M_n = \int_0^1 x^n \, d\mu_F(x).$

Case $n=0$ is trivial: $M_0 = \int_0^1 d\mu_F(x) = 1.$

Note also that we have $\int_0^{1/3} d\mu_F(x) = \frac{1}{2}.$

Case $n=1$ we solve by splitting the integral in two parts: $$ M_1 = \int_0^1 x \, d\mu_F(x) = \int_0^{1/3} x \, d\mu_F(x) + \int_{2/3}^1 x \, d\mu_F(x) $$ The first term we rewrite using scaling: $$ \int_0^{1/3} x \, d\mu_F(x) = \{ x = \frac{1}{3} y \} = \int_0^1 (\frac{1}{3}y) \, d\mu_F(\frac{1}{3}y) = \int_0^1 \frac{1}{3}y \, \frac{1}{2} d\mu_F(y) = \frac{1}{6} \int_0^1 y \, d\mu_F(y) = \frac{1}{6} M_1 $$ and the second term using reflection: $$ \int_{2/3}^1 x \, d\mu_F(x) = \int_{1/3}^0 (1-z) \, d\mu_F(1-z) = \int_0^{1/3} (1-z) \, d\mu_F(z) \\ = \int_0^{1/3} d\mu_F(z) - \int_0^{1/3} z \, d\mu_F(z) = \frac{1}{2} - \frac{1}{6} M_1 $$ Thus, $$ M_1 = \frac{1}{6} M_1 + \left( \frac{1}{2} - \frac{1}{6} M_1 \right) = \frac{1}{2}. $$

Case $n=2$ is solved in the same way: $$ M_2 = \int_0^1 x^2 \, d\mu_F(x) = \int_0^{1/3} x^2 \, d\mu_F(x) + \int_{2/3}^1 x^2 \, d\mu_F(x) $$ where $$ \int_0^{1/3} x^2 \, d\mu_F(x) = \int_0^1 (\frac{1}{3}y)^2 \, d\mu_F(\frac{1}{3}y) = \int_0^1 \frac{1}{9}(y)^2 \, \frac{1}{2}d\mu_F(y) = \frac{1}{18} \int_0^1 y^2 \, d\mu_F(y) = \frac{1}{18} M_2 $$ and $$ \int_{2/3}^1 x^2 \, d\mu_F(x) = \int_{1/3}^0 (1-z)^2 \, d\mu_F(1-z) = \int_0^{1/3} (1-z)^2 \, d\mu_F(z) \\ = \int_0^{1/3} d\mu_F(z) - 2 \int_0^{1/3} z \, d\mu_F(z) + \int_0^{1/3} z^2 \, d\mu_F(z) = \frac{1}{2} - 2 \cdot \frac{1}{6} M_1 + \frac{1}{18} M_2. $$ Thus, $$ M_2 = \frac{1}{18} M_2 + \left( \frac{1}{2} - 2 \cdot \frac{1}{6} M_1 + \frac{1}{18} M_2 \right) = \frac{1}{9} M_2 + \frac{1}{2} - 2 \cdot \frac{1}{6} \cdot \frac{1}{2} = \frac{1}{9} M_2 + \frac{1}{3} $$ so $$\frac{8}{9} M_2 = \frac{1}{3}$$ i.e. $$M_2 = \frac{9}{8} \cdot \frac{1}{3} = \frac{3}{8}.$$