What's the idea of an action of a group?

I'd say originally the action of a group on a set is what motivates many groups. Thus the symmetric group $S_n$ is in fact defined by its action on the set $\{1,\ldots ,n\}$. Or the symmetry group of an object (a regular dodecahedron, say) is inherently given by the way how this (abstract) group acts on the (more concrete) object. Often one gets insights out of this, for example the symmetry group $G$ of the cube operates on the 6-element set of faces, the 8-element set of vertices, the 12-element set of edges, the 4-element set of spacial diagonals, the 3-element set of axes and the 2-element set of inscribed tetrahedra, each giving rise to interesting facts about $G$.

Given arbitrary $G$ and $M$, you of course always have the trivial action at least. Without knowing more about $G$ and $M$, it is hard to say whether there are any more. For example if $G$ is cyclic of prime order $p$ and $M$ has less than $p$ elements, there is indeed only the trivial action. If there is no interesting relation between $G$ and $M$ that can be exploited, a nontrivial action might be defined in a very "dull" way (say, by an explicit multiplication table), but that won't give much insight into the group and set in question.

Let's say we have a set of interesting things; for example, we might have the set $C$ of faces of a cube, or the set $\mathbb{R}^3$ of points in Euclidean 3-space.

To any set $S$, there is a group denoted $\mathsf{Aut}(S)$ which consists of all bijective functions $f:S\to S$, where the operation is $\circ$ (composition), and the identity is just the identity function $\mathrm{id}_S(s)=s$.

However, it's usually not the case that we care about every possible way of rearranging (i.e., permuting) the elements of our set; for example, the permutation of $C$ swapping two adjacent sides and doing nothing else is not really as natural a thing to consider as the permutation of $C$ that a rotation of the cube induces. And there are tons of crazy ways of permuting $\mathbb{R}^3$ that can't even be written down; but we're mostly interested in mapping $\mathbb{R}^3$ to itself in geometrically interesting ways, like reflection or rotation.

Thus, we decide to restrict our attention to a certain subgroup $G$ of $\mathsf{Aut}(S)$.

That is (in essence) a group action: we have a group $G$, each of whose elements determines a permutation of $S$.

In the case of $\mathbb{R}^3$, we might decide in advance that the only permutations we are interested in are bijective linear transformations from $\mathbb{R}^3$ to itself, since after all $\mathbb{R}^3$ has this nice vector space structure, and we might as well preserve it. Then we are doing representation theory; we have a group $G$, each of whose elements is assigned a linear isomorphism from $\mathbb{R}^3$ to itself, which (thought of as a matrix) is an element of $\mathsf{GL}_3(\mathbb{R})$.

Now, sometimes we start out with the group $G$ already in mind. Maybe it came up in a problem we are doing, and we noticed that there is a natural way that its elements can give rise to permutations of some set (or linear isomorphisms of some vector space). Then, what we have is not a subgroup $G\subset \mathsf{Aut}(S)$ or $G\subset\mathsf{GL}_n(\mathbb{R})$, but instead, a homomorphism $\rho:G\to\mathsf{Aut}(S)$ or $\rho:G\to\mathsf{GL}_n(\mathbb{R})$.

Note that there is a phenomenon that can occur in this more general case, that does not occur in the way I initially described things, that deserves to be mentioned: multiple group elements can correspond to the same permutation (or linear isomorphism). In other words, the homomorphism $\rho:G\to\mathsf{Aut}(S)$ might have a non-trivial kernel.