Prove that $O(n)$ is a maximal compact subgroup of $GL(n,\mathbb R)$
Solution 1:
Let $K$ be a compact subgroup of $GL(n,\mathbb R)$. Let $\beta:\mathbb R^n\times\mathbb R^n\to\mathbb R$ be any inner product and consider the function $\bar \beta:\mathbb R^n\times\mathbb R^n\to\mathbb R$ such that $\bar \beta(x,y)=\int \beta(kx,ky)\,\mathrm dk$, where $\mathrm dk$ is the normalized Haar measure on $K$ (this makes sense because the function $k\in K\mapsto \beta(kx,ky)$ is continuous. One can easily check that $\bar\beta$ is an inner product on $\mathbb R^n$ and that the action of $K$ on $\mathbb R^n$ preserves $\bar\beta$. This means that $K$ is conjugated to a subgroup of $O(n,\mathbb R)$.
Solution 2:
Let $P$ be symmetric positive. Then there is an orthogonal $Q \in O(n)$, such that $Q^tPQ = \Lambda$ is diagonal. We have $Q^tP^kQ = \Lambda^k$ for every $k\in\mathbb Z$, hence $(\Lambda^k)$ has also bounded norm, by $C'$, say. But now, for every diagonal entry $\lambda$ of $\Lambda$ with unit eigenvector $e$: \[ \lambda^k = \|{\Lambda^ke}\| \le C'\|e\|, \quad k \in \mathbb Z \] As $\lambda$ is real and positive, this implies $\lambda = 1$. So $\Lambda = I_n$ and hence $P = I_n$.
Regarding the question in the title: Let $A \in GL(n,\mathbb R) \setminus O(n)$, then there is some $x$ with $\| x \| = 1$ and $\|Ax\| \ne 1$. Let $Q \in O(n)$ with $Qx = \frac{Ax}{\|{Ax}\|}$. Then $Q^tA$ maps $x$ to $\|Ax\|x$, hence $Q^tA$ has $\|Ax\|$ as an eigenvalue. Since $\|Ax\| \ne 1$, arguing as above, no compact subgroup of $GL(n,\mathbb R)$ can contain $Q^tA$. So no compact subgroup containig $O(n)$ can contain $A$ and we are done.