Why a holomorphic function satisfying these conditions has to be linear?

Consider the family $\mathscr{F} = \{ f^n : n \in \mathbb{Z}^+\}$, where $f^n$ denotes the $n$-fold iterate of $f$, $f\circ f \circ \dotsc \circ f$.

Listen to what Montel has to say about that family. And assume, for the sake of contradiction, that $f$ were not linear.


Since $\Omega$ is bounded, $\mathscr{F}$ is a normal family. To simplify notation, let us assume that $z_0 = 0$. Then in a neighbourhood of $0$, we have the Taylor expansion

$$f(z) = z + \sum_{k=2}^\infty a_k z^k.$$

If we already know that all $a_k$ for $2 \leqslant k < m$ are zero, iterating the expansion $f(z) = z + a_m z^m + O(z^{m+1})$ leads to

$$f^n(z) = z + n\cdot a_m z^m + O(z^{m+1}),$$

which is proved by induction,

$$\begin{align} f^{n+1}(z) &= f(f^n(z))\\ &= f^n(z) + a_m(f^n(z))^m + O(f^n(z)^{m+1})\\ &= z + n\cdot a_m z^m + O(z^{m+1} + a_m(z + O(z^m))^m + O(z^{m+1})\\ &= z + (n+1)a_m z^m + O(z^{m+1}). \end{align}$$

In other words, we have

$$\left(\frac{d}{dz}\right)^m \left(f^n\right)\bigl\lvert_{z = 0} = n\cdot f^{(m)}(0)$$

for $m \geqslant 2$ if we already know that $f^{(k)}(0) = 0$ for $2\leqslant k < m$. But the family of $m^{\text{th}}$ derivatives of a normal family is again normal, so $\left(\left(\frac{d}{dz}\right)^m \left(f^n\right)\bigl\lvert_{z = 0}\right)_{n\in \mathbb{N}}$ must have a convergent subsequence. By the above, that is only possible if $f^{(m)}(0) = 0$.

Thus all derivatives of order $> 1$ of $f$ vanish in $0$, and $f(z) = z$ follows.