Finite groups of functions under function composition
Depending on exactly what your requirements are, there does not exist such a function. First let's suppose you're working with rational functions $f(z) = \frac{p(z)}{q(z)}$ where $z$ is a complex parameter and $p, q$ have no common roots. In order for $f$ to be invertible by another rational function, it needs to be a bijective function from the Riemann sphere to itself. Now:
- If $\deg p > 1$ then $f^{-1}(0)$ consists of more than one point, and
- If $\deg q > 1$ then $f^{-1}(\infty)$ consists of more than one point.
It follows that in order for $f$ to be invertible we must have $f(z) = \frac{az + b}{cz + d}$; moreover we need $ad - bc \neq 0$, and this is sufficient.
The resulting group is very well-understood. It is the group of Möbius transformations or, equivalently, the projective special linear group $\text{PSL}_2(\mathbb{C})$, and all of its finite subgroups are known. Remarkably, they are described by Dynkin diagrams (see e.g. this MO question). Explicitly they are:
- the finite cyclic groups,
- the finite dihedral groups,
- the tetrahedral group,
- the octahedral group, and
- the icosahedral group.
The good news is that there are elements of order $5$. The bad news is that they cannot be realized over $\mathbb{Q}$.
To see this, note that $\text{PSL}_2(\mathbb{C})$ admits a 2-to-1 homomorphism $\text{SL}_2(\mathbb{C}) \to \text{PSL}_2(\mathbb{C})$; thus if $f(z) = \frac{az + b}{cz + d}$ (written so that $ad - bc = 1$) is an element of order $5$ or $7$, its preimages in $\text{SL}_2(\mathbb{C})$, one of which is given by the matrix $$M = \left[ \begin{array}{cc} a & b \\\ c & d \end{array} \right],$$
must have order $5, 7, 10$ or $14$. Since it has determinant $1$, its eigenvalues must be a primitive root of unity $\zeta_n$ (where $n = 5, 7, 10, 14$) and $\zeta_n^{-1}$. This means that its trace must be $$\zeta_n + \zeta_n^{-1} = 2 \cos \frac{2 \pi i}{n}$$
but it must also be $a + d$, which is rational. Since $\zeta_n + \zeta_n^{-1}$ is a sum of algebraic integers, it is an algebraic integer, hence is rational if and only if it is an integer. But $2 \cos \frac{2 \pi i}{n}$ is an integer if and only if it's equal to $2, 1, 0, -1, -2$, and this gives $n = 1, 2, 3, 4, 6$. Thus we cannot have $n = 5, 7, 10, 14$ or any number greater than $6$ for that matter.
The above argument is closely related to the crystallographic restriction theorem.
You can look for a rotation in $\mathbb{H^2}$. For example suposse fixed $i$ and then $-i$. The expression looking for $w=w(z)$ with $$\frac{w+i}{w-i}=k\frac{z+i}{z-i}$$ Then $$w(z)=\frac{(k+1)z+(k-1)i} {(k-1)z+(k+1)i}$$ You want $w^5(z)=z$. In matrix notation you need that $$\left( \begin{array}{ll} k+1 & k-1\\ k-1 & k+1 \end{array}\right)^5$$ be a scalar matrix. As $$\left( \begin{array}{ll} k+1 & k-1\\ k-1 & k+1 \end{array}\right)^5=\left( \begin{array}{ll} 16 k^5+16 & 16 k^5-16\\ 16 k^5-16 & 16 k^5+16 \end{array}\right)$$ Then $k^5=1$, for example $k=e^{2\pi i/5}$. If you want $w^7(z)=z$, in the same way $k^7=1$.
Certainly, it is not a simply and closed expression with integer numbers.