Does there exist a nontrivial rational function which satisfies $f(f(f(f(x))))=x$?

It's pretty well known and easy to show that for a linear fractional function $f(x)=\frac{ax+b}{cx+d}$ to be an involution, ie. $f(f(x))=x$, then $a\cdot d=-1$ is a necessary and sufficient condition.

An example of a rational function which satisfies $f(f(f(x)))=x$ is $\frac{x-1}{x}$.

I was fooling around with a few of these types of questions with a friend, and we got stuck on $f(f(f(f(x))))=x$. Does there exist a solution $f(x)=\frac{g(x)}{h(x)}$ where $g(x), h(x) \in \mathbb{Z}[x]$, and $f(f(x))\neq x$.

So far we've discovered that $f$ is bijective, $F(x):=f(f(x))$ is an involution, and $f(x)=\frac{1+ix}{x+i}$ works if we loosen the requirement on integer/rational coefficients, as $f(f(x))=\frac{1}{x}$ in this case.

We highly suspect that if $g(x), h(x)$ exist, then they are degree 3 or above as $F(F(x))=\frac{ax+b}{cx-a}$ doesn't seem to be yielding any acceptable solutions (and degree 2 doesn't satisfy bijectivity).

Consider the following

$$f(x)=\frac{1+x}{1-x}$$ $$\implies ff(x)=-\frac 1x$$ $$\implies fff(x)=\frac{x-1}{x+1}$$ $$\implies ffff(x)=x$$

Postscript... I was thinking along the lines of $$f(x)=\tan(\arctan x+\frac{\pi}{4})$$

It is easily verified that this is the same function as I have given above.

Furthermore we can fairly easily check that if $$f(x)=\tan(\arctan x+\frac{\pi}{3})$$ then $$fff(x)=x$$ although this function does not fit the requirements.

Therefore consider the function $$f(x)=\tan(\arctan x+\frac{\pi}{n})$$ where $n$ is prime. We can postulate that $$f^{(n)}(x)=x$$ but such a function would also not satisfy the requirements.

The only reason why this works "nicely" for $n=4$ is because $\tan\frac{\pi}{4}\in\mathbb{Q}$

Suppose $f$ is a Mobius transform of order $n> 1$.

Since we are in characteristic zero, $f$ can't be conjugate to a translation, and since $n > 1$ it can't be the identity, so $f$ must have two distinct fixed points in $\Bbb P^1(\Bbb C)$, and then the derivative of $f$ at those fixed points must be a primitive $n$th root of unity.

Moreover, the product of those derivative is always $1$, so the sum of the two derivatives has to be $2\cos(2k\pi/n)$ where $k$ is coprime with $n$ (this is in fact equivalent to $f$ having order $n$).

But, the sum of the derivatives at the fixed points is a rational expression in terms of the coefficients of the Mobius transform (it is a pretty big one, you should convince yourself this works because it is a symmetric algebraic expression in terms of the two fixed points, and those fixed points satisfy a degree $2$ equation whose coefficients are expressions in terms of the coefficients of $f$)

So if such an $f$ exists with coefficients in $K$, you must have $2\cos(2\pi/n) \in K$. If $K = \Bbb Q$ this implies $\Bbb Q(2\cos(2\pi/n)) = \Bbb Q$. Looking at the degree of those extensions you get $\phi(n) = 1$ or $2$, so $n=2,3$ or $4$

To make a suitable Mobius transform of order $4$, take for example the one whose fixed points are at $\pm i$ and where the derivative are $\pm i$ there.

Then letting $g(z) = (z-i)/(z+i)$, we must have that $g(f(g^{-1}))(z) = iz$ (after conjugating by $g$, its fixed points are $0$ and $\infty$ so it is a scaling).

Thus $f(z) = g^{-1}(ig(z)) = (1+z)/(1-z)$

Since every Moebius Transform is associated to an invertible matrix $$ f(z) = {{az + b} \over {cz + d}}\quad \Leftrightarrow \quad H = \left( {\matrix{ a & b \cr c & d \cr } } \right) $$ and the convolution of two Transformations is associated to the product of the associated matrices $$ f_{\,2} \circ f_{\,1} \quad \Leftrightarrow \quad H_{\,2} \;H_{\,1} $$ then $$ f \circ f \circ f \circ f = id\quad \Leftrightarrow \quad H^{\,4} = I $$ corresponds to find the $4$-th roots of the Identity matrix.

Standing the above, as the "basic" fourth roots we can take: $$ \left( {\matrix{ 1 & 0 \cr 0 & 1 \cr } } \right),\;\;\left( {\matrix{ 1 & 0 \cr 0 & { - 1} \cr } } \right),\;\;\,\left( {\matrix{ 1 & 0 \cr 0 & i \cr } } \right) $$ noting that the $1$st is the square of the $2$nd, which in turn is the square of the $3$rd.
So, only for the $3$rd the square is different from $I$.

Also, it is to be noted that the rotation matrix $$ \left( {\matrix{ {\cos \left( {\pi /2} \right)} & { - \sin \left( {\pi /2} \right)} \cr {\sin \left( {\pi /2} \right)} & {\cos \left( {\pi /2} \right)} \cr } } \right) \equiv - i\,\left( {\matrix{ 1 & 0 \cr 0 & i \cr } } \right) $$ can be derived from the third, so in case we can replace one for the other.

As rightly underlined by Quasicoherent, the correspondence $f \Rightarrow H$ is net of a common factor, so that at the end we shall write $$ H^{\,4} = I\quad \Rightarrow \quad f = {{\lambda \,h_{\,1,1} \,z + \lambda \,h_{\,1,2} } \over {\lambda \,h_{\,2,1} \,z + \lambda \,h_{\,2,2} }}\quad \left| {\;\lambda \in \mathbb C} \right. $$