Differentiable functions satisfying $f'(f(x))=f(f'(x))$

Well, I see this has already been answered, but I have something to add about how one might additional construct solutions.

In particular, I looked at construction of analytic solutions, which will work on $\mathbb{C}$ as well as in $\mathbb{R}$. Let's suppose that $f(x)$ has a fixed point at $c$, so $f(c) = c$. By the functional equation, $f(f'(c)) = f'(f(c)) = f'(c)$, so $f'(c)$ is also a fixed point of $f$, and by the same logic we have the sequence $f'(f'(c)), f'(f'(f'(c))) ...$ are all fixed points of $f$. This is not nice. To simplify, let's further impose the condition that $f'(c) = f(c) = c$. Then we can look at $f$ locally around $c$, which allows us to look at $f$ as a Taylor series around $c$. If we differentiate $f'(f(x)) = f(f'(x))$, we obtain the equation: $$ f''(f(x))f'(x) = f''(x) f'(f'(x)) $$ which at $x = c$ tells us $f''(c) f'(c) = f''(c) f'(c)$, which tells us nothing about $f''(c)$. Let's just call $\lambda = f''(c)$. We can repeat this process, because in fact $c$ and $\lambda$ uniquely determine $f'''(c)$, as long as $c\ne0,1$: $$ \begin{align} f'''(f(x)) f'(x)^2+f''(f(x))f''(x)&=f'''(x)f'(f'(x))+f''(x)^2 f''(f'(x))\\ f'''(c) c^2+\lambda^2 &=f'''(c)c+\lambda^3\\ f'''(c) &=\frac{\lambda^3 - \lambda^2}{c^2 - c} \end{align} $$ We can similarly derive $$ f^{(4)}(c) = \frac{(4\lambda^2 - 3\lambda(c+1))(\lambda^3 - \lambda^2)}{(c^3 - c)(c^2 - c)} $$ and so on. In general, using Faa di Bruno's formula, we will have: $$ \sum_{k=1}^n f^{(k+1)}(c)B_{n,k}(f'(c),f''(c),...,f^{(n - k + 1)}(c)) = \sum_{k=1}^n f^{(k)}(c)B_{n,k}(f''(c),f'''(c),...,f^{(n - k + 2)}(c)) $$ where $B_{n,k}$ are the incomplete Bell polynomials. These equations turn out to be linear in $f^{n+1}(c)$, and as long as $|c| \ne 0,1$ will always have a unique solution in terms of the lower derivatives, so thus $c$ and $f''(c) = \lambda$ uniquely determine a Taylor series for a function solving $f'(f(x)) = f(f'(x))$: $$ f(x) = c + c(x-c) + \lambda\frac{(x-c)^2}{2} + \left(\frac{\lambda^3 - \lambda^2}{c^2 - c} \right)\frac{(x-c)^3}{6} + \left(\frac{(4\lambda^2 - 3\lambda(c+1))(\lambda^3 - \lambda^2)}{(c^3 - c)(c^2 - c)}\right)\frac{(x-c)^4}{24} + ... $$ Of course, that's limited to cases where you have $c \in \mathbb{C}\setminus\{z\text{ | } |z| = 0,1\}$ such that $c = f(c) = f'(c)$. But this is actually pretty general; all the previous analytic solutions are in fact special cases except for $f(x) = \frac{1}{x}$. The solution $f(x) = n^{-n+1} x^n$ corresponds to the choice of $c = n$ and $\lambda = n-1$. The solution $f(x) = \alpha e^x$ corresponds to $c = \lambda = - W(-\alpha)$, where $W(z)$ is some branch of the Lambert W function (note that this isn't necessarily real, which is why I chose to stick to $\mathbb{C}$ instead of $\mathbb{R}$). The linear solutions correspond to $\lambda = 0$, and the quadratic proposed by Cameron Williams corresponds to $c = 1 + i$, $\lambda = 1$. We can parametrize the solution functions, so let $f_{c,\lambda}(x)$ be the unique solution with $f(c) = f'(c) = c$ and $f''(c) = \lambda$. In this case, $\lim\limits_{t\rightarrow -1} f_{t,t-1}(x) = \lim\limits_{t\rightarrow -1} t^{-t + 1} x^t = \frac{1}{x}$, so this is almost a solution of this type.

This question seems like it will not have a simple solution.

As others have pointed out, there are a couple of interesting solutions based on , such as $f(x) = 0$ or $f(x) = e^x$ or $f(x) = (-1)^{n+1}n^{n+1}x^{-n}$ or $f(x) = 1/x$ (Though technically the last one isn't a function from $\mathbb{R}$ to $\mathbb{R}$).

However, what hasn't been done so far in the comments is to characterize the solutions by looking at the various values that $f \circ f'$ can take.

Therefore let's define $g = f \circ f' = f' \circ f$. Now the reason that makes this problem seem like it may not have a simple solution, is the fact that there are already a whole bunch of solutions just for the case $g = 0$, not all of which are smooth.

For $g = 0$, a trivial solution would be the function $f = 0$. However, if we take a function $h$ with compact support whose derivative is bounded, we can consider the function $f(x) := h(x - T)$ for some $T \in \mathbb{R}$. If we set $T$ large enough, $f$ takes on zero for all values of $f'$, satisfying $f \circ f' = 0$. Similarly, we know that $f$ is bounded given that it is continuous, leading us to conclude that $f' \circ f = 0$ for large enough T. This means that by taking any differentiable function with compact support and a bounded derivative, we can shift it far enough to the right (or left) to obtain a function that satisfies the equation of the question just for the special case of $g=0$. If I'm not mistaken, there are even functions in $C^1(\mathbb R)\setminus C^2(\mathbb{R})$ with finite support and a bounded derivative, meaning that we have some quite "not-nice" solutions.amongst the set of solutions.

Of course it isn't as easy to find solutions for other values of $g$ as it is for $g = 0$, but I think that the fact that this special case already has such a wide variety of solutions highlights the fact that this problem may be quite difficult to solve.