How to show a representation is irreducible?

Let $G$ be a group with $n$ elements, and let $\chi_R(g)$ be the trace of the matrix for $g$ in the representation $R$. So we have a vector for $R$: $$(\chi_R(g_1),\ldots,\chi_R(g_n))$$ which I'll denote just $\chi_R$. Now define an inner product on these vectors: for any representations $R$ and $S$, $$\langle \chi_R,\chi_S\rangle = \frac{1}{n}\sum_{g\in G}\chi_R(g)\overline{\chi_S(g)}$$ One of the big results about characters is that for any two inequivalent irreducible representations, $\langle \chi_R,\chi_S\rangle = 0$, and for any irreducible $R$, $\langle \chi_R,\chi_R\rangle = 1$. This is known as the orthogonality of irreducible characters.

Now suppose a representation $T$ is reducible. To keep notation simple, let's do the case where $T$ is the direct sum of two irreducible representations $R$ and $S$ (possibly equivalent). It's not hard to see that then $\chi_T = \chi_R+\chi_S$. By linearity of the inner product and the orthogonality just mentioned, if $R$ and $S$ are inequivalent $$\langle \chi_T,\chi_T\rangle = \langle \chi_R+\chi_S,\chi_R+\chi_S\rangle = 1+1 = 2$$ On the other hand, if $R$ and $S$ are equivalent, then $\chi_R = \chi_S$ (another big result about characters), so $\chi_T=2\chi_R$ and $$\langle \chi_T,\chi_T\rangle = 4\langle \chi_R,\chi_R\rangle = 4$$ The same argument works if $T$ is a sum of more than two irreducible characters, the notation just gets a bit messier.

So irreducible characters are characterized by $\langle \chi_R,\chi_R\rangle = 1$.

Here are a couple of examples. First, the simplest reducible case: the trivial 2-dim representation of the group with one element, $G=\{1\}$. If $V$ is the vector space, with basis (say) $\{v_1,v_2\}$, then $V$ is the direct sum $[v_1]\oplus[v_2]$ (I'm using [] for "the span of", so $V=[v_1,v_2]$). Now $[v_1]$ and $[v_2]$ are obviously invariant subspaces, so we have an invariant direct-sum decompositon, which makes the representation reducible. This reducibility shows up in the matrix for the sole element of $G$, which is $$\left[\begin{align*}\fbox{1} & 0 \\ 0 & \fbox{1}\end{align*}\right]$$ The boxes are the matrices in the 1-dim representation of $G$. Notice the block form: $$\left[\begin{align*}\fbox{$M_{11}$} & 0 \\ 0 & \fbox{$M_{22}$}\end{align*}\right]$$ Writing $T$ for the 2-dim rep, $\langle \chi_T,\chi_T\rangle = 4$. Notice that this a special case of the $\chi_T=2\chi_R$ situation described above.

Second example: let $G=S_3$. The 2-dim rep for $G$ comes from treating $G$ as the symmetry group of an equilateral triangle: each of the symmetries can be realized by an invertible linear transformation of the plane. (They also happen to be isometries.) Let's say the elements of $G$ are $\{1,r,r^2,f_1,f_2,f_3\}$ where $r$ is a $120^\circ$ rotation and $f_1,f_2,f_3$ are the three "flips" (reflexions). The vector $\chi_T$ looks like this: $(2, -1, -1, 0, 0, 0)$. (I leave it as an exercise to show that these entries are correct.) Now we compute: $$\langle \chi_T,\chi_T\rangle = \frac{1}{6}\left(2^2+(-1)^2+(-1)^2+0^2+0^2+0^2\right) = 1$$ So the representation $T$ is irreducible, which is pretty easy to see directly.

I'm not familiar with Hill's book, but I would imagine it has the results about characters I mentioned. There is a set of on-line notes I'd recommend: Finite Groups .