What is the meaning of an "irreducible representation"?
A representation of the group $G$ means a homomorphism from $G$ into the group of automorphisms of a vector space $\mathbf{V}$. Essentially, you are trying to interpret each element of $G$ as an invertible linear transformation $\mathbf{V}\to\mathbf{V}$, in order to try to understand the group $G$ by how it "acts on $\mathbf{V}$."
If you have an action $\rho_1$ of $G$ on a vector space $\mathbf{W}$ (that is, one representation), and you have some other action $\rho_2$ of $G$ on another vector space $\mathbf{Z}$ (another representation), then you can use these two actions to construct an action of $G$ on the vector space $\mathbf{W}\oplus\mathbf{Z}$: just let $G$ act on the first coordinate using the old action on $\mathbf{W}$, and let it act on the second coordinate using the old action on $\mathbf{Z}$.
The point to observe, however, is that the action of $G$ on $\mathbf{W}\oplus \mathbf{Z}$ defined this way does not give you any new insights into the structure of $G$: anything you can glean about $G$ from this action, you can learn about $G$ by considering the original actions $\rho_1$ and $\rho_2$. So this new action does not give us anything new.
Conversely, suppose you have one representation $\rho$, with $G$ acting on $\mathbf{V}$, and that there are proper subspaces $\mathbf{W}$ and $\mathbf{Z}$ of $\mathbf{V}$ that satisfy the following properties:
- $\mathbf{V}=\mathbf{W}\oplus\mathbf{Z}$; and
- The action of every $g\in G$ on $\mathbf{V}$ maps $\mathbf{W}$ to itself; and
- The action of every $g\in G$ on $\mathbf{V}$ maps $\mathbf{Z}$ to itself.
Then you can look at the restriction of the action of $G$ on $\mathbf{W}$ to get a representation, and the restriction on $\mathbf{Z}$ to get another representation; and these two representations will give you all the information from the original representation, the same way we had before. The advantage being that since $\mathbf{W}$ and $\mathbf{Z}$ are proper subspaces of $\mathbf{V}$, they have smaller dimension and, presumably, it's easier to understand a subgroup of linear automorphisms for them than for $\mathbf{V}$.
So the moral is that we want to find representations that cannot be "broken up" into smaller ones, because there's no point in trying to understand ones that do break up, we can focus our attention on those that don't, because all the other representations can be built up in terms of the ones that cannot be broken up.
The irreducible representations are precisely the ones that cannot be broken up into smaller pieces (at least for finite groups). Maschke's Theorem says that if you have a representation $\rho$ of a finite group $G$ acting on $\mathbf{V}$, and $\mathbf{W}$ is a subspace of $\mathbf{V}$ such that for all $g\in G$, the image of $\mathbf{W}$ under the action of $g$ is $\mathbf{W}$ itself, then you can find a subspace $\mathbf{Z}$ of $\mathbf{V}$ such that $\mathbf{V}=\mathbf{W}\oplus\mathbf{Z}$ and every $g\in G$ maps $\mathbf{Z}$ to itself (that is, in order to break up $\rho$ into two smaller pieces, it is enough to find a single proper piece on which $\rho$ acts; then you can find a complement for it).
With this in mind, we say:
Let $\rho\colon G\to \mathrm{Aut}(\mathbf{V})$ be a representation of $G$. We say that $\rho$ is irreducible if and only if $\mathbf{V}$ is not the zero vector space, and the only subspaces of $\mathbf{V}$ that are mapped to themselves under the action of every $g\in G$ are $\{\mathbf{0}\}$ and $\mathbf{V}$ itself.
An irreducible representation of $SO(3)$ will be a representation of $SO(3)$ that is irreducible. $SO(3)$ acts naturally on the vector space $\mathbb{R}^3$: it consists of all automorphisms of $\mathbb{R}^3$ that respect the inner product, so this is itself a representation of $SO(3)$ (which is irreducible, because no proper subspace of $\mathbb{R}^3$ is sent to itself by all elements of $SO(3)$).
(Maschke's Theorem holds if the vector space is over a field of characteristic $0$, or if the characteristic does not divide the order of the group; there are similar theorems for certain kinds of infinite groups, but it does not hold for arbitrary infinite groups in general.)
if $\rho:G\to{\rm GL}_n(\mathbb{C})$ is a finite dimensional complex representation of the group $G$, then $\rho$ is irreducible if whenever $\rho(G)(V)\subseteq V$ then $V=\mathbb{C}^n \text{ or } 0$ (there are no non-trivial subspaces of $\mathbb{C}^n$ left invariant by $\rho(G)$).
here's a "visual" example. the real representation of ${\rm SO}_3(\mathbb{R})$ (the elements acting on $\mathbb{R}^3$ by rotations about the origin) is irreducible (there is no line or plane in $\mathbb{R}^3$ that is invariant under all rotations).