Why are compact operators 'small'?

It may help to think of the special case of diagonal operators, that is, elements of $\ell_\infty $ acting on $\ell_2$ by multiplication. Here compact operators correspond to sequences which tend to 0, "are infinitesimally small". This is a commutative situation, in which everything reduces to multiplication of functions. So, general compact operators can be called noncommutative infinitesimals.

A shorter explanation, but with less content: every ideal in a ring can be thought of as a collection of infinitesimally small elements, because they are one step (quotient) away from being zero.

Finite rank operators are small (in that they squish a large space into a small one). In a Hilbert space (or more generally, a Banach space with the approximation property, which includes most familiar examples), compact operators are precisely the operator-norm limits of finite rank operators. That's what I think of when I hear that statement.

Alternatively, a compact operator from $X$ to $Y$ squishes the unit ball of $X$ (which is "big") into a compact subset of $Y$ (which, in Banach spaces, is typically "small" in that it has empty interior).