How to find $f$ if $f(f(x))=\frac{x+1}{x+2}$
Solution 1:
You can easily check that if we define
$$f_{A} = \frac{ax+b}{cx+d} \quad \text{for} \quad A = \begin{pmatrix}a & b \\ c & d \end{pmatrix}, $$
then $f_{A} \circ f_{B} = f_{AB}$. Thus any matrix $A$ satisfying
$$ A^{2} = k \begin{pmatrix} 1 & 1 \\ 1 & 2 \end{pmatrix}, \quad \text{for some} \ k \neq 0 $$
gives rise to a solution. Mathematica yields two different solutions
$$ f(x) = \frac{1}{x+1} \quad \text{and} \quad f(x) = \frac{2x+1}{x+3}, $$
but I'm not sure if other solutions exist.
Solution 2:
As sos440 mentioned already, linear fractional transforms correspond to matrices.
Using that, we can simplify the problem by using conjugation.
$$ (g \circ f \circ g^{-1}) \circ (g \circ f \circ g^{-1}) (x) = g\left(\frac{g^{-1}(x)+1}{g^{-1}(x)+2} \right)$$
Here, we use also linear fractional transform $g$.
The RHS of the equation above corresponds to a matrix that is conjugate to $$\pmatrix{{1}&{1}\\{1}&{2}}$$
This matrix is diagonalizable with distinct real positive eigenvalues $\lambda_1, \lambda_2$. Let $k= \lambda_1/\lambda_2 >0$.
The matrix that corresponds to $g^{-1}$ is the matrix $X^{-1}=\pmatrix{{v_1} & {v_2}}$ where $v_1$, $v_2$ are eigenvectors corresponding to eigenvalues $\lambda_1$, $\lambda_2$ respectively. Thus, $g$ corresponds to the matrix $X=\pmatrix{{v_1} & {v_2}}^{-1}$. Then $$ X \pmatrix{{1}&{1}\\{1}&{2}}X^{-1}= \pmatrix{{\lambda_1}&{0}\\{0}&{\lambda_2}} $$ so that $$ g\left(\frac{g^{-1}(x)+1}{g^{-1}(x)+2} \right) \mbox{ corresponds to the diagonal matrix } \pmatrix{{\lambda_1}&{0}\\{0}&{\lambda_2}} $$
After conjugation, our problem becomes a simpler problem finding solution to $$ F\circ F (x) = kx $$ where $F=g\circ f\circ g^{-1}$.
This new problem looks similar to the following problem: $$G\circ G(x) = x$$
The solution to this one is not only $x$, $-x$, but also $$G(x)=\cases{{-\frac{1}{2}x \mbox{ if $x<0$}}\\{-2x \mbox{ if $x\geq 0$}}}$$
Similarly if we let $F$ as follows, $$F(x)=\cases{{-\frac{\sqrt{k}}{2}x \mbox{ if $x<0$}}\\{-2\sqrt{k}x \mbox{ if $x\geq 0$}}}$$
then $F\circ F (x) = kx$.
For this $F$, we recover $f$ by reversing the conjugation $f= g^{-1}\circ F\circ g$. This resulting function $f$ will not be one of the two functions given.
Solution 3:
Suppose $f(x)=\frac{ax+b}{x+c}$. Then $$ f(f(x))= \frac{(a^2+b)x+b(x+c)}{(a+c)x + b+c^2}=\frac{x+1}{x+2} $$ yields the equations $$ \frac{a^2+b}{a+c}=1, \frac{b(a+c)}{a+c}=b=1, \frac{b+c^2}{a+c}=2, \frac{a^2+1}{a+c}=1$$ from which we find $$c=a^2-a+1$$ and hence $$a^4-2a^3+a^2-2a=a(a^3-2a^2+a-2)=0$$ This last equation has but two real solutions, $a=0$ and $a=2$, corresponding to the two solutions given by sos440: $$ f(x)=\frac{1}{x+1} \mbox{ and } f(x)=\frac{2x+1}{x+3}.$$
Added: The last equation has $a=\pm i$ as roots, but these yield the degenerate functions $f(x)=i$ and $f(x)=-i$ which do not yield the required $f(f(x))$.
Solution 4:
If one is looking only for solutions that are fractional-linear, $f(z)=(az+b)/(cz+d)$, then the two that @sos440 has found are the only ones, as suggested by the interesting response of @i707107.
Here’s a somewhat more elaborate response than that of @MatthewConroy, his answer really disposes of the question fully. The fractional linear transformations with real coefficients may be looked at as transformations of the real projective line, $\mathbb P^1(\mathbb R)$ and equally well as transformations of the complex line, $\mathbb P^1(\mathbb C)$. Those that are unequal to the identity fall into three classes, the elliptic, without fixed points on the real line, such as $(z+1)/(-z+1)$, which permutes the four points $\{\infty, -1, 0, 1\}$ cyclically; the parabolic, which have a single fixed point that in some sense is of multiplicity two, such as $z\mapsto z+1$, whose only fixed point is $\infty$; and the hyperbolic, with two fixed points.
Our given map $z\mapsto(z+1)/(z+2)$ is of this last type, and its fixed points are $\rho>0$ and $\rho'<0$, the two roots of $X^2+X-1$. As you see, they satisfy $\rho+\rho'=\rho\rho'=-1$, and it happens that $\rho$ is the reciprocal of the “Golden Ratio”, but that is just an accident here. Now, working in the group of complex fractional-linear transformations, we may conjugate $\rho'$ to $0$ and $\rho$ to $\infty$, and ask about the transformations that leave leave these two fixed. Of course these are all $z\mapsto\lambda z$, we’re just talking multiplication here. In particular, if we have a transformation of this type that takes a particular $\zeta$ to $\lambda\zeta$, then there are just two transformations that square to this, namely $z\mapsto\mu z$ for the two values of $\mu$ with $\mu^2=\lambda$.
By hand and with the help of a symbolic program, I conjugated $(\infty,0)$ to $(\rho,\rho')$, and the transformation that corresponds to $z\mapsto\lambda z$ is $$ \pmatrix{1+(1-\rho)\lambda&-\rho+\rho\lambda\\-\rho+\rho\lambda&1-\rho+\lambda } \>\colon\>z\mapsto\frac{\bigl(1+(1-\rho)\lambda\bigr)z-\rho+\rho\lambda } {(-\rho+\rho\lambda)z + 1-\rho+\lambda}\,. $$ You can check that $\lambda_-=-2-\rho$ gives $1/(z+1)$, $\lambda_+=2+\rho$ gives $(2z+1)/(z+3)$, and $\Lambda=\lambda_+^2$ gives $(z+1)/(z+2)$.
Solution 5:
$f$ is a bijection from $\Bbb R \setminus \{-2\}$ to $\Bbb R \setminus \{1\}$.
Strictly speaking, it cannot have a functional square root : If $f = g \circ g$, then $g(g(-5/3)) = f(-5/3) = -2$. If $g(-5/3) = -2$ then $g(-2)=-2$ and we get that $g\circ g$ is defined at $-2$ when it should not. So $g(-5/3) \neq -2$ and $f$ is defined at $g(-5/3)$ : $f(g(-5/3)) = g(-2)$. Again, if $f(g(-5/3)) = -2$ then $g(-2) = -2$, so $f$ is defined at $f(g(-5/3))$, and so $g$ has to be defined at $g(-2)$. In any case, $g(g(-2))$ exists while it shouldn't.
To make the question interesting we plug the hole and consider $f$ as a bijection from $X = \Bbb R \cup \{\infty \}$ to itself. $f$ has two fixpoints $0 < x_1 < x_2$, any iterate of $f$ is still an homography so still has only those two same fixpoints. This means that you can partition $X' = X \setminus \{x_1,x_2\}$ into sequences of the form $\{f^k(x), k \in \Bbb Z\}$
If you don't need $g$ to be continuous, there are lots of ways to do define a functional square root. Given any point $x \in X'$, you can choose $g(x)$ to be any point $y \in X'$ as long as $y \notin \{f^k(x), k \in \Bbb Z\}$. This choice will determine $g$ at all the $f^k(x)$ and $f^k(y)$ for $k \in \Bbb Z$. Repeat this until you have defined $g$ on all of $X'$. Finally define either $g(x_i)= x_i$ or $g(x_i) = x_{3-i}$.
If you want $g$ to be continuous, there are two cases to consider. $X'$ has two connected components, $(x_1;x_2)$ and $(x_2; \infty) \cup \{\infty\}\cup\{- \infty ; x_1\}$, which should just be called $(x_2 ; x_1)$. Those two components are stable by $f$. It is useful to put an order on those components such that $f$ is increasing. On the first component it is the standard order, but on the other one, we have $y < \infty < z$ if $y > x_2$ and $z < x_1$.
If $g$ switches the connected components $(x_1;x_2)$ and $(x_2 ; x_1)$ of $X'$, $g$ will be determined by its image on an interval of the form $[x ; f(x))$ where $x \in X'$ : you have to send $x$ to some $y$ (on the other component), send $f(x)$ to $f(y)$, and in-between, $g$ can be any strictly increasing (for the orders defined above) continuous function.
If $g$ doesn't switch them, we have to define $g$ separately on each component. To do that, pick an $x \in X'$, choose some $g(x) \in (x ; f(x))$. Then we must have $g(g(x)) = f(x)$, and we can pick any strictly increasing continuous function on $[x ; g(x)]$. This determines $g$ completely on the component of $x$. do the same on the other component and we are done.
In both cases, for $g$ to be continuous, we must have $g(x_i) = x_i$ if $x_i$ is one of the two fixpoints.
Also, those constructions can give you functional square roots that are smooth on $X'$ (I(m not sure what happens at the ficpoints) and are still not one of the two homographies that you have found.