How can there be multiple irreducible representations of a group each having distinct dimension?

The matrix elements for the $(2l+1)$-dimensional irreducible representation of SO(3) are given by:

$D_{m',m}^l(\phi_1,\Phi,\phi_2)=[i^{m'-m}\sqrt{(l+m')!(l-m')!(l+m)!(l-m)!}$ $\times(\frac{1+\cos\Phi}{2})^l (\frac{1-\cos\Phi}{1+\cos\Phi})^{(m'-m)/2}$ $\times\sum\limits_k \frac{(-1)^k}{(l+m-k)!(l-m'-k)!(m'-m+k)!k!}$ $\times(\frac{1-\cos\Phi}{1+\cos\Phi})^k] e^{im'\phi_2}e^{im\phi_1}$

My question is, how can there be multiple irreducible representations of SO(3) each of which has a different dimension? I thought that by definition an irreducible representation is one which has no non-trivial invariant subspace. I know that SO(3) can be represented by the group of 3x3 symmetric matrices of determinant = 1 (I thought this had something to do with the fact that it takes 3 parameters to describe a 3D rotation). If I have a larger matrix (e.g. 5x5) doesn't it have to have a non-trivial invariant subspace (e.g. of dimension 2), and therefore be reducible?

The application here is the expansion of functions using generalized spherical harmonics, which is accomplished (per the Peter-Weyl theorem) using an infinite linear combination of the matrix elements of the irreducible representationS (emphasis on the S) of SO(3).

$\mathrm{SO}(3)$ is a wonderful group, but it quickly got bored rotating vectors in space. It wanted to be used to help solve differential equations, and so desperately wanted to act on functions. Then one day, it realized it could act on functions by acting on their input.

For instance, it contains a 90 degree clockwise rotation in the $x$-$y$ plane. It takes $(x,y,z)$ and replaces it with $(y,-x,z)$.

Well, the function $2x+3y+7z$ can also be rotated! We just replace $x$ by $y$, $y$ by $-x$, and leave $z$ alone: $2x+3y+7z$ is rotated into $2y+3(-x)+7z = -3x + 2y + 7z$. If we consider the action on all these linear $Ax+By+Cz$ it becomes clear that $\mathrm{SO}(3)$ acts by 3×3 matrices on the vector space with basis $\{x,y,z\}$.

Yay, $\mathrm{SO}(3)$ can act on functions now, and it acts on those linear functions as 3×3 matrices.

What about quadratics? Well we could have $2x^2 + 3xy + 7y^2$. That same 90 degree rotation takes it to $2(y)(y) + 3(y)(-x) + 7(-x)(-x) = 7x^2 - 3xy + 2y^2$. Yay, now $\mathrm{SO}(3)$ acts as 6×6 matrices on the quadratic polynomials $Ax^2+Bxy+Cy^2+Dxz+Eyz+Fz^2$ spanned by $\{ x^2, xy, y^2, xz, yz, z^2 \}$.

It turns out though that $\mathrm{SO}(3)$ doesn't swirl these functions around very thoroughly. If you take a function, and rotate its input, all it does is rotate the laplacian. In particular, notice that it takes any harmonic polynomial (one with 0 laplacian) to a harmonic polynomial. For quadratic polynomials, this is particularly easy to describe:

When you use $\mathrm{SO}(3)$ to rotate $Ax^2+Bxy+Cy^2+Dxz+Eyz+Fz^2$, it leaves the laplacian, $A+C+F$, alone. So it will never rotate $x^2$ into $xy$. In fact, it leaves the 5-dimensional space spanned by $\{ x^2-y^2, y^2-z^2, xy, xz, yz \}$ invariant. If we write things in these coordinates, then we get $\mathrm{SO}(3)$ acting as 5×5 matrices. In fact it acts irreducibly.

What happened to the 6th dimension? Well, it is spanned by a very silly function, the sphere: $x^2+y^2+z^2$. If you rotate the sphere, you get the sphere. On this one dimensional space, $\mathrm{SO}(3)$ is represented by 1×1 matrices. Well, actually, matrix. Every single rotation in $\mathrm{SO}(3)$ acts as the identity matrix $[1]$, since every rotation leaves the sphere alone.

If we use colors, like pink, lime, and periwinkle to describe vectors, then a polynomial like $5x^2-5y^2 + 6x + 3$ is composed of a pink term, $5x^2-5y^2$, where $\mathrm{SO}(3)$ acts in a 5×5 manner, a lime term, $6x$, where $\mathrm{SO}(3)$ acts in a 3×3 manner, and a periwinkle term, $3$, where $\mathrm{SO}(3)$ acts in a 1×1 manner.

A polynomial like $7x^2+5y^2$ is a little trickier to see its colors (it is a good thing $\mathrm{SO}(3)$ is so clever), $7x^2+5y^2 = 4(x^2+y^2+z^2) + 3(x^2-y^2) + 4(y^2-z^2)$. The fist term $4(x^2+y^2+z^2)$ is periwinkle where $\mathrm{SO}(3)$ acts in a 1×1 manner, but the next two terms $3(x^2-y^2) + 4*(y^2-z^2)$ are pink where $\mathrm{SO}(3)$ acts in a 5×5 manner.

That's the most confusing way to write down the irreducible representations of $\text{SO}(3)$ I've ever seen. Here's a much less confusing one: the irreducible representation of $\text{SO}(3)$ of dimension $2n+1$ is precisely its action on the space of harmonic polynomials in three variables $x, y, z$ of degree $n$ (given by extending its action on $x, y, z$ multiplicatively). Maybe this will help you play with these representations and see why they don't behave the way you think they do.

Dear okj, in the $SO(3)$ case, and perhaps at least morally even more generally, you may imagine the higher-dimensional irreducible representations as tensors. In particular, the 5-dimensional representation is a symmetric traceless tensor $T$. It satisfies $$T_{ij} = T_{ji}, \quad \sum_{i=1}^3 T_{ii} = 0 $$ The first condition reduces 9 elements of the matrix to 6 independent ones; the vanishing of the trace reduces 6 to 5. Now, if an $SO(3)$ matrix $M$ acts on vectors via $$ \vec v \to M \vec v ,$$ then it acts on $T$ via $$ T \to M T M^{T}.$$ Note that it preserves the symmetry of $T$ as well as its vanishing trace. However, it's equally clear that there is no linear combination of the five entries in $T$ that would be invariant under all $SO(3)$ transformations. So there is no singlet, and because $3+1+1$ and $1+1+1+1+1$ are the only possible ways how to decompose a 5-dimensional representation to representations you know, it follows that none of them works and there is no triplet (vector), either.

More general representation of $SO(3)$ may also be written as tensors with some constraints. The constraints are needed to make them irreducible.

For more general groups, both finite and Lie groups, the more complicated representations are not that easily written as tensors, but in principle, it's always possible to find any irrep in the tensor product of "fundamental representations".

Here is an important point that may be confusing you. One may build more complicated representations of a group as $$ V_1\oplus V_2$$ which is manifestly reducible but one may also build them as $$ V_1\otimes V_2, $$ the tensor product. This is like creating object with many indices. There is no simple link between the direct sum and the tensor product. The tensor product typically contains completely new, higher-dimensional ireducible representations than $V_1$ and $V_2$.