Prove that there exists no differentiable real function $g(x)$ such that $g(g(x))=-x^3+x+1$.

Solution 1:

We have $$ g(g(g(x)) = -g(x)^3 + g(x) + 1 \iff \\ g(-x^3+x+1) = -g(x)^3 + g(x) + 1 $$ For $x=1$ this turns into $$ g(1) = -g(1)^3 + g(1) + 1 \iff \\ g(1)^3 = 1 $$ So $g(1) = 1$, if $g$ is a real valued function.

Differentiating both sides of $g(g(x)) = -x^3+x+1$ gives $$ g'(g(x))\, g'(x) = -3x^2 + 1 $$ This gives $$ g'(g(1))\,g'(1) = - 2 \iff \\ g'(1)^2 = -2 $$ which is not possible for real valued $g'(x)$ and thus for $g(x)$, as the derivative of a real valued function is a real valued function.

Solution 2:

Recall that a fixed point of a function $f$ is a solution to $f(x) = x$.

The only fixed point of $g \circ g$ is $1$, so the only fixed point of $g$, if any, is also $1$, as any fixed point of $g$ is also a fixed point of $g \circ g$. If $g$ has no fixed point but instead interchanges $1$ and $\xi = g(1) \neq 1$, then the graph of $g$ must connect $(\xi, 1)$ and $(1, \xi)$. These points lie on opposite sides of the line $y=x$ regardless of the value of $\xi$, so $g$ has a fixed point not at $1$ by the Intermediate Value Theorem, a contradiction.

Therefore, $g$ has one fixed point, at $1$. Taking derivatives of $g \circ g$ gives $g'(x) g'(g(x)) = 1 - 3x^2$. Setting $x = g(x) = 1$ gives $g'(1)^2 = -2$, impossible. The conclusion follows.