Why do we define modules to be over *abelian* groups?

Aside from the observation that has already been made in other comments that the other module axioms already force a module to be abelian, let me make the following philosophical remarks.

Rings naturally want to act on abelian groups, in the following sense: you can make sense of the notion of monoid $M$ in any monoidal category $(V, \otimes)$, and once you have a monoid one very natural notion of an action of that monoid is another object $S$ in $V$ and an action map $M \otimes S \to S$ satisfying certain axioms. For example:

• If $V = (\text{Set}, \times)$, then a monoid in $V$ is a monoid (e.g. a group) in the usual sense and a monoid action is a set on which the monoid acts in the usual sense.
• If $V = (\text{Top}, \times)$, then a monoid in $V$ is a topological monoid (e.g. a topological group, e.g. a Lie group) and a monoid action is a topological space on which the monoid acts continuously.
• If $V = (\text{Ab}, \otimes)$, then a monoid in $V$ is a ring (exercise!) and a monoid action is a module over the ring in the usual sense.

So there's a fairly general definition of what it means for a monoid to act on something, and applied to rings-as-monoids-in-$\text{Ab}$ it reproduces the usual notion of module.

Actually there is another very general thing we can do. We can make sense of what it means for a monoid in $V$ to act on an object of any category which is enriched over $V$: namely, if $M$ is a monoid in $V$ and $C$ is a $V$-enriched category, then a monoid action on an object $S$ in $C$ is a morphism $M \to \text{End}(S)$ of monoids (in $V$). For example:

• If $V = (\text{Set}, \times)$, then a $V$-enriched category is a (locally small) category, so monoids can act on objects in essentially any category.
• If $V = (\text{Ab}, \otimes)$, then a $V$-enriched category is a preadditive category (e.g. an abelian category), so rings can act on objects in any such category. Examples include categories of chain complexes or sheaves of abelian groups.

The first construction becomes a special case of the second construction if $V$ is closed monoidal; in this case specifying an action map $M \otimes S \to S$ is the same as specifying a map $M \to [S, S]$, where brackets denote internal hom, and the axioms required of an action map are equivalent to requiring that this map $M \to [S, S]$ is a morphism of monoids.

The monoidal category of abelian groups under tensor product is closed monoidal, so the above remarks apply to it. In particular, if $A$ and $B$ are any two abelian groups, then the set $\text{Hom}(A, B)$ of homomorphisms between them naturally acquires an abelian group structure. This means that if $A$ is an abelian group, then $\text{End}(A)$ naturally has an abelian group structure in addition to a second monoid structure given by composition; in fact, it's naturally a ring, the universal ring acting on $A$. This doesn't happen for not-necessarily-abelian groups.

The category of abelian groups is really something special all on its own. It is not just a convenient subcategory of the category of groups; it really has its own structure that the category of groups fails to have. (Groups have their own structure, though: the category of groups can be enriched over the category of groupoids. But that's another story.)

Let $R$ be a ring with $1$. If $M$ is a group, written multiplicatively, equipped with an $R$-action, then, for any $x,y\in M$, $(xy)^2 = (1+1)\cdot (xy) = [(1+1)\cdot x][(1+1)\cdot y] = x^2 y^2$. So $yx=xy$, and $M$ is abelian.

If $R$ doesn't have a $1$, then we conclude only that $rM$ is abelian for all $r\in R$. In this case, we might be able to define noncommutative modules, but they are likely to be very awkward. For example, let $R=\{0,\epsilon\}$ with $\epsilon+\epsilon=\epsilon^2=0$, which is the simplest non-unital ring. Then a noncommutative $R$-module can be described by a group $G$, a normal subgroup $N\lhd G$, and an abelian subgroup $A \leq N$ isomorphic to $G/N$ (the quotient $G\to G/N$ is the multiplication map $\epsilon\cdot: G\to \epsilon G$). Understanding the ways this can happen is very subtle, probably intractable, and certainly only tangentially related to the study of $R$.

Another perspective: a module is just a quotient of a free module. So if we want noncommutative modules, then we want to consider, for example, $R * R$, the amalgamated product, to be a module over $R$. But $r(R*R)=(rR)*(rR)$ is never abelian unless $rR=0$, so this is only an $R$-module when $R$ has trivial multiplication.

In short, addition should always be commutative. If I really wanted to think of a non-abelian group as a module over something, I would probably work over a semigroup $S$, and define an $S$-module to be a group $G$ and a semigroup homomorphism $S\to\operatorname{End}(G)$ (one way to define an $R$-module is an abelian group $M$ with a homomorphism of rings $R\to \operatorname{End}(M)$).

Proposition. Let $R$ denote a unital ring. Suppose $M$ is an $R$-module whose underlying group is not necessarily abelian. Then $M$'s underlying group is abelian.

Proof. Consider $x,y \in M$. Then $2(x+y) = 2x+2y$, where $2$ is shorthand for $1_R+1_R$. So $x+y+x+y = x+x+y+y$. By cancellation, $y+x=x+y$.

So if we're going to consider non-abelian modules, then we should also drop the condition that $a(x+y) = ax+ay$ for all $a \in R$ and all $x,y \in M$. Once we've done this, there are lots of examples of such things. For instance, any group, denoted additively, becomes a non-abelian $\mathbb{Z}$-module as follows.

$$nx = \underbrace{x +\cdots + x}_n, \qquad 0x = 0, \qquad (-1)x = -x$$

This satisfies all the module axioms we're used to, except for $n(x+y) = nx+ny$ which clearly does not hold. Observe, in particular, that the axiom $(a+b)x = ax+bx$ still holds.

Anyway, since we've dropped one of our "distributivity" laws, namely $a(x+y)=ax+ay$, it probably makes sense to allow $R$ to be an arbitrary near-ring.