Exercise on representations

By hypothesis, for all $h \in H$ and basis $(e,f)$ one has $$\det(h(e)-e,h(f)-f) =0.$$

Let $g \in H$ different from the identity.

$\bullet$ Suppose that $1$ is the only eigen-value of $g$. Then in some basis $(e,f)$ the matrix of $g$ is $Mat(g) = \left( \begin{smallmatrix} 1&1\\0&1 \end{smallmatrix}\right)$. Let $h \in H$, then $$(1) \quad \det(h(e)-e,h(f)-f)=0,$$ $$(2) \quad \det(hg(e)-e,hg(f)-f)=\det(h(e)-e,h(e)+h(f)-f) = 0.$$ If $h(e)-e=0$, then $h(e)$ is colinear to $e$. If $h(f)-f=0$, then (2) shows that $h(e)$ is colinear to $e$. If $h(e) \neq e$ and $h(f)\neq f$, then $h(e)+h(f)-f$ is colinear to $h(e)-e$ (by (2)). The latter is colinear to $h(f)-f$ (by (1)). This implies that $h(e)$ is colinear to $e$. Hence $ke$ is fixed by $H$ and we are done.

$\bullet$ Suppose that $g$ has two distinct eigenvalues $1$ and $a$. Then is some basis $(e,f)$ : $Mat(g) = \left( \begin{smallmatrix} 1&0\\0&a \end{smallmatrix}\right)$.

Lemma: if $h \in H$ then (i) $h(e)=e$ or (ii) $h(f)$ is colinear to $f$.

Proof: We have $$\det(h(e)-e,h(f)-f) = 0,$$ $$\det(hg(e)-e,hg(f)-f) = \det(h(e)-e,a h(f)-f) = 0.$$ If $h(e) \neq e$, then $h(f)-f$ and $a h(f)-f$ are colinear to each other (because they are both colinear to $h(e)-e$). Since $a \neq 1$, this implies that $h(f)$ is colinear to $f$. QED

To conclude we have to show that either every $h \in H$ satisfies (i) or every $h \in H$ satisfies (ii). If not, then let $h \in H$ not satisfying (i) and $k \in H$ not satisfying (ii). The matrices of $h$ and $k$ have the forms $Mat(h) = \left( \begin{smallmatrix} 1&0\\\alpha&* \end{smallmatrix}\right)$ and $Mat(k) = \left( \begin{smallmatrix} 1&\beta\\0&* \end{smallmatrix}\right)$ with $\alpha \neq 0$ and $\beta \neq 0$. We check that $\det(hk-{\rm{Id}}) = -\alpha \beta$ which is a contradiction.