Calculate the cohomology group of $U(n)$ by spectral sequence.
Solution 1:
I'll expand on the hint given by Robert. $U(1)$ is just $S^1 \subset \mathbb{C}$, and its cohomology ring is the exterior algebra $\Lambda[c_1]$. Here, the subscript of the generator indicates its degree.
For $n = 2$, we have, as you described, a fibre bundle $U(1) \to U(2) \to S^3$. Thus there is a Serre spectral sequence with \begin{equation} E_2^{p,q} = H^p(S^3,\mathcal{H}^q U(1)) \cong H^p(S^3) \otimes H^q(U(1)) \Rightarrow H^{p+q}(U(2)). \end{equation}
We have $E_2^{p,q} \cong \Lambda[c_1,c_3]$. By lacunary reasons, this spectral sequence collapses on the second page, and so we deduce $H^*(U(2)) \cong \Lambda[c_1,c_3]$.
In general, the spectral sequence for the fiber bundle $U(n-1) \to U(n) \to S^{2n-1}$ always collapses on the second page, and you can use induction to prove the proposition $H^*(U(n)) \cong \Lambda[c_1,c_3,\ldots,c_{2n-1}]$.
I like Mimura and Toda's The topology of Lie groups as a reference for these types of calculations. If I recall, they also discuss how to obtain the above proposition using Morse theory (or maybe just a rational decomposition of $U(n)$).
You also asked about $O(n)$, which is trickier. First, $O(n)$ is not path-connected, but has two path components each homeomorphic to $SO(n)$, so we may as well consider $SO(n)$ instead. There is a section in Hatcher (3.D) that is dedicated to computing the mod 2 cohomology of $SO(n)$, using a cell decopmosition.
You can also compute the mod 2 cohomology of $O(n)$ and $SO(n)$ using spectral sequences, but the problem here is that the analogous spectral sequences need not collapse, so the analysis is more subtle. One can instead induct on Stiefel manifolds, which generalize $O(n)$. I learnt how to do this from May et al's notes on characteristic classes. In particular, see Theorem 2.4 for the cohomology groups of $SO(n)$, although the ring structure requires consideration of Steenrod operations. May also discusses cohomology away from the prime 2.