Order of matrices in $GL_2(\mathbb{Z})$

Let's assume that $A\in GL_2\left(\mathbb{Z}\right)$ has finite order, let say $n$. Then $A^n=I$. Now look at this matrix as a matrix in $M_2(\mathbb Q)$: we have $\det A=\pm 1$ and if consider its minimal polynomial $m_A\in\mathbb Q[X]$ this must divide $X^n-1$ and $\deg m_A\le 2$.

If $\deg m_A=1$, then $m_A=X-1$, that is, $A=I$ and $n=1$ or $m_A=X-1$, that is, $A=-I$ and $n=2$. (The only rational roots of unity are $\pm1$.)

If $\deg m_A=2$, then $m_A(X)=(X-\epsilon_1)(X-\epsilon_2)$, $\epsilon_1\neq\epsilon_2$, where $\epsilon_i$ is an $n$th root of unity. Since $m_A\in\mathbb Q[X]$ we have $\epsilon_1+\epsilon_2\in\mathbb Q$ and $\epsilon_1\epsilon_2\in\mathbb Q$. In fact, $m_A(X)=X^2-\operatorname{tr}(A)X\pm1$, and thus we get $\epsilon_1+\epsilon_2\in\mathbb Z$ and $\epsilon_1\epsilon_2=\pm1$. If $\epsilon_1=1$, then $\epsilon_2=-1$, $m_A=X^2-1$ and therefore $A^2=I$, so $n=2$. Now suppose that $\epsilon_i\neq 1$. Some easy remarks show that the only case to consider is $\epsilon_1\epsilon_2=1$, that is, $\epsilon_2=\overline{\epsilon}_1$. Then $m_A(X)=X^2-2\cos\frac{2k\pi}{n}X+1$, with $n\ge 2$ and $1\le k<n$. But $2\cos\frac{2k\pi}{n}\in\mathbb Z$ implies $2\cos\frac{2k\pi}{n}=0,\pm1,\pm2$ so we have five possible minimal polynomials: $X^2+1$, $X^2\pm X+1$ and $X^2\pm2X+1$. (In fact only three, since the last two can't appear.) These give: $A^2+I=0$, so $n=4$; $A^2+A+I=0$, so $n=3$ and $A^2-A+I=0$, so $n=6$.

I leave the OP the pleasure to find for each $n\in\{1,2,3,4,6,\infty\}$ a matrix $A\in GL_2\left(\mathbb{Z}\right)$ of order $n.$

Suppose $A\in\mathrm{GL}_2(\mathbb Q)$ has order $n$. Clearly, there is a degree-$2$ extension $\mathbb Q(\sqrt d)/\mathbb Q$ such that $A\in\mathrm{GL}_2(\mathbb Q(\sqrt d))$ is diagonalizable, say $PAP^{-1}=\begin{bmatrix}a & c\\&b\end{bmatrix}$ for $P\in\mathrm{GL}_2(\mathbb Q(\sqrt d))$.

Since this matrix has order $n$ (conjugation doesn't change the order), we must have $a^n=b^n=1$. Thus, the multiplicative group $\langle a,b\rangle\subseteq \mathbb Q^\times$ is $\langle\zeta_n\rangle$, so we must have $\mathbb Q(a,b)=\mathbb Q(\zeta_n)\subseteq\mathbb Q(\sqrt d)$. However, since $\mathrm{Gal}(\mathbb Q(\zeta_n)/\mathbb Q)\cong(\mathbb Z/n)^\times$, the extension $\mathbb Q(\zeta_n)/\mathbb Q$ has degree $\varphi(n)$. Thus, we obtain $\varphi(n)=[\mathbb Q(\zeta_n):\mathbb Q]\le[\mathbb Q(\sqrt d):\mathbb Q]=2$.

To summarize, we need $\varphi(n)\le2$, which can only happen for $n\in\{1,2,3,4,6\}$.

Hint: consider the minimal polynomial for such a matrix. If $A$ has order $n$, the minimal polynomial must divide $x^n-1$ and no smaller $x^m-1$.

Longer answer. Requires knowledge of cyclotomic polynomials.

If $A$ is a $2\times 2$ integer matrix such that $A^n=I,$ then consider the characteristic $$p(x)=x^2-(\operatorname{tr} A)x+\det A$$

It has integer coefficients, and since $A^n=I,$ this means $\det A=\pm 1.$

This also means the minimal polynomial for $A$ is either $p(x)$ or $x+1$ or $x-1.$

Now, if the minimal polynomial is $x-1,$ then $A=I$ and the minimal $n$ is $1.$

If the minimal polynomial is $x+1,$ then $A=-I,$ so the minimal $n$ is $2.$

If the minimal polynomial is $p$ then $p(x)\mid x^n-1.$

If $p$ is not irreducible, by the rational root theorem, $p$ is either $x^2-1,(x+1)^2,$ or $(x-1)^2.$

If $x^2-1,$ then $n=2$ is the minimum.

If $(x+1)^2$ or $(x-1)^2,$ this would mean $x^n-1$ has repeated roots, which it doesn’t. So the minimal polynomial for $A$ can’t be $(x\pm 1)^2$ if $A^n=I.$

If $p$ is irreducible, then since:

$$x^n-1=\prod_{d\mid n} \Phi_d(x)$$ where $\Phi_d$ are the cyclotomic polynomials, and are known to be irreducible monic integer polynomials.

Thus $p(x)= \Phi_d(x)$ for some $d.$

But we also know that $\deg \Phi_d =\phi(d),$ the Euler $\phi$ function.

So this means $\phi(d)=2.$

But $\Phi_d(x)\mid x^d-1,$ so this means $A^d=I.$

So now you just need to find all $d$ such that $\phi(d)=2.$

I’ll skip that part. It is relatively basic to get that $d=3,4,6.$

Then we’d like to find specific examples.

$$A_3=\begin{pmatrix}-1&1\\-1&0\end{pmatrix},\\ A_4=\begin{pmatrix}0&1\\-1&0\end{pmatrix},\\ A_6=\begin{pmatrix}1&1\\-1&0\end{pmatrix}$$