Solve $f(x)f(2x^2) = f(2x^3+x)$
Solution 1:
As observed in other answers, the leading coefficient of $f$, and the constant term both have to equal $1$ (if $f$ is not identically $0$). Now assume that $f$ has complex root with absolute value greater than 1. Let $z$ one of them with greatest absolute value. Now $2z^3 + z$ is also a root, by triangle inequality $$ |2z^3 + z| = |z||2z^2 + 1| \geq |z|(|2z^2| - 1) > |z|(2 - 1) = |z|, $$ contradicting the assumption. So all the roots have absolute value $\leq 1$. But the product of the absolute values of roots is $1$ (the constant term is $1$), so all of them are of absolute value one.
Again using the previous inequality with $|z| = 1$ instead of $|z| > 1$, we get that for all the roots $$ |2z^2 + 1| = |2z^2| - 1, $$ which is quite easily only possible if $z^2 = -1$ so $z = \pm i$. So only possible roots are $\pm i$ and if $i$ is, so is $2i^3 + i = -i$ and vice versa.
Lastly, to ensure that multiplicities of $i$ and $-i$ are equal: if $f$ has roots, it has factor $z^2 + 1$ by our observation, and factoring that out we see that the resulting polynomial also satisfies the functional equation. So indeed, only solutions are constant polynomials $0$ and $1$ and $(x^2+1)^k$ for positive integer $k$.
Solution 2:
Another partial result: either $f(x) = 0 $ for all $x$ or $f$ has no real roots.
Proof:
Let $r$ be the largest real root of $f$. Then, by the functional equation, $ 2r^3+r$ is also a root of $f$, and so by the definition of $r$, $$ r \geq 2r^3+r $$ which simplifies to $$ 0 \geq r$$ Similarly, if we let $n$ be the smallest root of $f$, once again by the functional equation $2n^3+n $ is also a root and so $$ n \leq 2n^3+n$$ which simplifies to $$ 0 \leq n$$ So any root of $f$ must be between $0$ and $0$ and hence can only be $0$. So now there are two possibilities: 1) $f$ has no real roots 2) $f$ only has roots at $x=0$. In the second case, $f$ must be of the form $$ f(x) = a x^n $$ for some constant $a$ and non-negative integer $n$. Substituting this in the functional equation yields $$ ax^n a(2x^2)^n = a (2x^3+x)^n$$ $$ a^2 2^n x^{3n} = a (2x^3+x)^n $$ which is only possible if $n=0$ since otherwise the RHS has multiple terms. Hence, we are left with $$ a^2 = a$$ and $$f(x)=a$$ so $a=0$ or $a=1$, but we must take $a=0$ since case 2) is under the assumption that $f$ has a root at $x=0$. This completes the proof.
Solution 3:
Not a solution, but some observations: If $f$ is constant, $f(x) = c$ then $c^2 = c$ therefore $c\in\{0,1\}$.
If $f$ is not constant, but has a root $a$, then (as gtrrebel said) $2a^3+a$ is a root too. If $a = 2a^3+a$, then $a=0$, that means $x$ is a factor of $f$. If $a\neq 2a^3+a$ then we $(x-a)(x-2a^3-a)$ is a factor of $f$. If $a$ is positive we can solve $a=2b^2$ for $b = \sqrt{a/2}$, then $2b^3+b$ is a root too with the same argument.
Solution 4:
This is a partial solution as well.
If $f$ has degree $n$, and leading coefficient $c$, then the leading coefficients of the equation are $c^22^n=c2^n$, so $c=1$.
If $f(0)=d$ then $d^2=d$. If $d=0$ then the equation shows that $f(x)$ is divisible by $x^3$; then repeat that and $f(x)$ is divisible by $x^9,x^{27}$ and so on. So $d=1$.
The coefficient of $x^{3n-1}$ is zero on the right-hand side, so it is zero on the left-hand side; so the coefficient of $x^{n-1}$ in $f(x)$ is zero.
$f(x)=x^2+1$ works. (so does f(x)=0, if that is a polynomial)