A representation of a finite group which is not completely reducible

Solution 1:

I was not able to find any reference for Maschke's theorem talking only about the finite dimensional representations.

The "modern" statement of Maschke's theorem (or at lest, the one ring theorists like) is this:

For any commutative ring $R$ and finite group $G$, the group ring $R[G]$ is semisimple iff $R$ is semisimple and the order of $G$ is a unit in $R$.

Semisimplicity, linked above, is the condition where all right $R$ modules split into sums of simple modules. This is the corresponding fact to the result in representation theory about all representations being completely reducible.

Now if $R$ is a field of characteristic zero, the order of $G$ is going to be a unit no matter what order it has, and furthermore, fields are semisimple. So in particular, you have a corollary:

For any field $F$ of characteristic 0 and finite group $G$, the group ring $F[G]$ is semisimple, and hence all of its modules (=representations of $G$) are completely reducible.

When the order of $G$ divides the characteristic of a field $F$, $F[G]$ does have representations that are not completely reducible. The easiest example in that case would have to be $F[G]$ itself, which necessarily has a nonzero Jacobson radical.

As a toy example, you could take the cyclic group of order two $C_2=\{1,c\}$ and the field $F_2$ of order two, and consider its group algebra $F_2(C_2)$. You get a ring of four elements $\{0,1,c,1+c\}$. You can see that $(c+1)^2=0$, so that $\{0,c+1\}$ is the nilradical (which is the Jacobson radical in this case).

Solution 2:

For a ring $R$, not necessarily commutative, but with a multiplicative identity, a non-zero $R$-module is a direct sum of simple modules if and only if it is a sum of simple modules, if and only if every submodule has a complement. A representation of a finite group $G$ over $\mathbf{C}$ is the same thing as a $\mathbf{C}[G]$-module. Every $\mathbf{C}[G]$-module $M$, whether finite dimensional or not, has the property that every submodule has a complement (the proof of Maschke's theorem does not use that the module is finite dimensional, only that $G$ is finite). So by the quoted result, every $\mathbf{C}[G]$-module is a direct sum of simple submodules, that is, every complex representation of $G$ is completely reducible.