How to solve the functional equation $ f(f(x))=ax^2+bx+c $
Solution 1:
SHORT ANSWER: A general solution to this problem is not known in the closed form, but some special cases can be solved. Sorry.
LONG ANSWER: Notice that if a function $f$ satisfies $$f(x)=(g^{-1} \circ h \circ g)(x)$$ for some $g,h$, then $$f^n(x)=(g^{-1} \circ h^n \circ g)(x)$$ where the superscript denotes functional iteration rather than exponentiation.
We can solve this equation for a whole class of quadratics $$f^2(x)=q(x)=ax^2+bx+c$$ for which $$c=\frac{b^2-2b}{4a}$$ This is because we can rewrite $$q(x)=ax^2+bx+\frac{b^2-2b}{4a}$$ as $$q(x)=a\bigg(x+\frac{b}{2a}\bigg)^2-\frac{b}{2a}$$ and so, by letting $g(x)=x-\frac{b}{2a}$ and $h(x)=ax^2$, $$q(x)=(g^{-1} \circ h \circ g)(x)$$ and thus $$q^n(x)=(g^{-1} \circ h^n \circ g)(x)$$ and since the formula for $h^n$ is $$h^n(x)=a^{2^n-1}x^{2^n}$$ we have $$q^n(x)=a^{2^n-1}\bigg(x+\frac{b}{2a}\bigg)^{2^n}-\frac{b}{2a}$$ and, finally, $$f(x)=q^{1/2}(x)=a^{\sqrt 2-1}\bigg(x+\frac{b}{2a}\bigg)^{\sqrt 2}-\frac{b}{2a}$$ So there's the solution for that special case. There is another special case when $$c=\frac{b^2-2b-8}{4a}$$ but the solution to that case is much longer, and so I will omit it and leave it to you for independent research.
There is another special case involving trigonometric functions. Note that if we let $$g(x)=\arccos x$$ $$h(x)=2x$$ we have $$(g^{-1}\circ h\circ g)(x)=\cos(2\arccos x)$$ and, using the double-angle formula, $$(g^{-1}\circ h\circ g)(x)=\cos^2(\arccos x)-\sin^2(\arccos x)$$ $$(g^{-1}\circ h\circ g)(x)=x^2-(1-x^2)$$ $$(g^{-1}\circ h\circ g)(x)=2x^2-1$$ and so $g^{-1}\circ h\circ g$ is a quadratic. Now let $$q(x)=(g^{-1}\circ h\circ g)(x)$$ so that $$q^n(x)=(g^{-1}\circ h^n\circ g)(x)$$ Now, since $$h^n(x)=2^n x$$ we have $$q^n(x)=\cos(2^n\arccos x)$$ and $$f(x)=q^{1/2}(x)=\cos(\sqrt{2}\arccos x)$$ which solves yet another special case. However, notice that this is only solved on the domain $[-1,1]$, because that is where $\arccos x$ is defined.
I am sure there are other special cases involving trigonometric functions, but I will leave those for you to find.
Solution 2:
This is only a partial answer, but it might be of some help.
If $f(x)=mx+d$, then $f(f(x))=m(mx+d)+d=m^2x+(m+1)d$, so any triple of the form $(0,b,c)$ with $b\ge0$ works, by taking $m=\sqrt b$ and $d=c/(1+\sqrt b)$.
Likewise, if $f(x)=m|x|^\sqrt2$, then $f(f(x))=m|(m|x|^\sqrt2)|^\sqrt2=m|m|^\sqrt2x^2$, so any triple of the form $(a,0,0)$ works, by taking $m=sgn(a)|a|^{1/(1+\sqrt2)}$.
So it looks to me like there are two natural questions: 1) Does $a\not=0$ force $b=c=0$? and 2) are there any triples with $b\lt0$?
Solution 3:
Here are some reflections for continuous and differentiable $f$. It turns out that there is a unique solution if some conditions are satisfied (you can look at the end for the final answer).
- $$ f(f(x))=ax^2+bx+c\implies f(ax^2+bx+c)=af(x)^2+bf(x)+c $$
- If $f(x)=f(y)$, $x\neq y$ then: $$ a(x+y)=b. $$ If $a(x+y)=b$ then in any case $ff(x)=ff(y)$ which means that either $f(x)=f(y)$ or $a(f(x)+f(y))=b$. In any case there are two non-equal numbers $x'$ and $y'$ such that $f(x')=f(y')$ and they lie on two different sides of $\frac{-b}{2a}$.
- If $f(x)=x$ then: $f(x)=ax^2+bx+c$ hence: $$ ax^2+(b-1)x+c=0\implies (b-1)^2\geq 4ac; x=\frac{-b+1\pm\sqrt{(b-1)^2-4ac} }{2a}. $$
- $f'(ax^2+bx+c)(2ax+b)=f'(x)(2af(x)+b)$ hence:
$$
x=\frac{-b}{2a}\implies f'(\frac{-b}{2a})=0\text{ or } f(\frac{-b}{2a})=\frac{-b}{2a}$$
On the other hand:
$$
f'(f(x)).f'(x)=2ax+b.
$$
So:
If $f'(x)=0$ then $x=\frac{-b}{2a}$.
Note that from large enough $x$ and $y$ satisfying $a(x+y)=b$ and $f(x)=f(y)$
- If $f(\frac{-b}{2a})=\frac{-b}{2a}$, then $f\circ f(\frac{-b}{2a})=\frac{-b}{2a}$ which means: $$ c-\frac{b^2}{4a}=-\frac b{2a}\implies b^2-4ac=2b. $$
moreover: $$ f'(f(\frac{-b}{2a})).f'(\frac{-b}{2a})=\implies f'(\frac{-b}{2a})=0. $$ If $b^2-4ac\neq 2b$, $f'(\frac{-b}{2a})=0$; so in any case:
$$ f'(\frac{-b}{2a})=0. $$
This means that the function is strictly increasing on the one side of $\frac{-b}{2a}$ and strictly decreasing on the other side.
Moreover if $f'(f(x))=0$ then $x=\frac{-b}{2a}$; so if there is $x$ such that $f(x)=\frac{-b}{2a}$ then $f'(f(x))=0$ which means that $x=\frac{-b}{2a}$. So:
$$f(\frac{-b}{2a})=\frac{-b}{2a}\implies b^2-4ac=2b$$
Without this condition there will be no answer.
- But if $a(x+y)=b$ and $f(x)\neq f(y)$, then $a(f(x)+f(y))=b$; but this means that both of them cannot be in the same time bigger (or smaller) than $\frac{-b}{2a}$; which is a contradiction since $f'(\frac{-b}{2a})=0$ and $\frac{-b}{2a}$ is an extremum point of $f$. Hence:
$$ f(x)=f(y), x\neq y\iff a(x+y)=b$$
Therefore it is enough to find the function for $x>\frac{-b}{2a}$.
- See that after using the information derived above:
$$
f(ax^2+bx+\frac{b^2-2b}{4a})=af(x)^2+bf(x)+\frac{b^2-2b}{4a}\implies\\
f\left(a(x+\frac{b}{2a})^2-\frac{b}{2a}\right)=a\left(f(x)+\frac{b}{2a}\right)^2-\frac{b}{2a}.
$$.
Note that if $a>0$, then $x>-\frac{b}{2a}\implies f(x)>-\frac{b}{2a}$; w.l.o.g. we assume $a>0$ and $x>-\frac b{2a}$. $$ af\left(a(x+\frac{b}{2a})^2-\frac{b}{2a}\right)+\frac{b}{2}=\left(af(x)+\frac{b}{2}\right)^2 $$ See that defining $g(x)=af(x)+\frac b2$, we get: $$ g(a(x+\frac{b}{2a})^2-\frac{b}{2a})=g(x)^2 $$ Define the function: $$ h(x)=\frac{\log g(x)}{\log(ax+\frac b2)}. $$ Then we get: $$ h(a(x+\frac{b}{2a})^2-\frac{b}{2a})=h(x) $$ This function can be shown to be a constant by constructing a decreasing sequence $u_n$ from an arbitrary $u$ and showing that $h(u)$ is equal to the limit of $u_n$ which is a constant value independent of $u$. Hence: $$ \frac{\log g(x)}{\log(ax+\frac b2)}=C\implies g(x)=(ax+\frac b2)^C\\ \implies f(x)=\frac 1a\left((ax+\frac b2)^C-\frac b2\right). $$ By plugging in to the original question, it turns out that $C=\sqrt 2$. Extending it for all $x$'s, we get:
$$ f(x)=\frac 1a\left(|ax+\frac b2|^{\sqrt 2}-\frac b2\right) \text{ if } b^2-4ac=2b. $$