What's so special about the group axioms?

I've only just begun studying group theory (up to Lagrange) following on from vector spaces and I am still finding them almost frustratingly arbitrary. I'm not sure what exactly it is about the axioms that motivated them defining groups.

My textbook asks you to list 'common features' of vector spaces and then later defines a set of axioms for vector spaces under addition, scalar multiplication and both, noting that the axioms under addition form an Abelian group. So are groups just a generalisation of vector spaces under any binary operation?

My main problem is that the book notes that 'the axioms may seem rather arbitrary' and links groups with vector spaces but doesn't elaborate.

When introducing groups, you are tasked to complete Cayley tables for the symmetries of an equilateral triangle and square. Then, similarly to the delivery of vector spaces, notes that the tables have common properties (Closure, Identity, inverse and association) and defines a group as a set of elements under a binary operation that has these features.

So what's so important about these 4 properties? For example, if 1 or 2 the properties were excluded form the axioms, or we added an extra few properties as axioms how would that cripple the effectiveness of groups?

Are the group axioms ever difficult to work with or do they always work, forgive the crude Littlewood analogy, like a mathematical skeleton key?

What is it about these 4 properties that make groups such a powerful tool in mathematics and physics?

My best guess is that a group is the best way to express our sense of symmetry and what is symmetric mathematically, but I would prefer some elaboration.

Yes, groups express symmetries of objects. If you want a symmetry to be a way to map an object into itself which preserves some property (say, location of vertices on a square, or distance between points in the plane,) and you want symmetries to be things that you can (a) chain together and (b) undo, then you've got a group. (As long as you also admit the trivial symmetry.) In this picture of symmetries as mappings, you don't really have to specify associativity, as it's a natural aspect of anything that looks like composition of functions (more precisely, of anything which fits into a category.)

You shouldn't think that groups are just generalizations of vector spaces. Anything that's a generalization from one case is a bad generalization! Groups are also generalizations of numbers (under addition and, in some cases, multiplication), symmetries of geometric objects (both discrete ones you mention and continuous ones,) and functions, under not only addition and multiplication but, critically, composition. An important historical example was the group of permutations of roots of a polynomial, which is used in the proof that quintic equations can't be solved in radicals and has led to huge areas of modern math.

As for loosening and strengthening some of the axioms: if you throw out inverses, you get monoids (semigroups, if you also throw out the identity.) These are extremely important objects in their own right, but they're too general to have a nice structure theory. The biggest problem is with quotients: surjective homomorphisms $G\to H$ naturally correspond to "normal subgroups" $K\subset G$, i.e. subgroups with $gKg^{-1}\subset K$ for all $g\in G$. There's no such simple way to characterize quotients $M\to N$ of monoids in terms of submonoids, because there's not necessarily a $g^{-1}$ for every $g$, so the most basic isomorphism theorems, which lay the foundation for everything you're likely to learn in elementary group theory, are no longer true. Thus we like to use groups instead of monoids when we can, and many actual things in the world come as groups, so we do so.

Strengthening the axioms does not "cripple the effectiveness" of groups so much as weakening them does. Various objects subject to axioms extending those of a group include abelian groups, vector spaces, modules, rings and algebras over rings, topological groups, Lie groups, and many more, and all of these are of great importance. But there are groups that are none of these things, (every group is in some sense topological, but we don't always want to think about the topology) so we start with just a plain group and strengthen the axioms in appropriate contexts.

At some level, these features are arbitrary - mathematicians study many objects which have more or less axioms than a group. They study abelian groups (adding the axiom $ab=ba$), monoids (which don't need inverses), loops (which don't need to be associative), and other objects. Groups, from a purely mathematical point of view, are fairly well-behaved objects - we don't need to put parenthesis everywhere (associativity), we're allowed to make cancellations (inverses, identity), and there's no magical, "oops, multiplying those two gives something entirely different" (closure). At some level, groups are interesting as a pretty general object for which we get many interesting results (like Lagrange's theorem or the classification of simple groups) which don't hold when we start to take away axioms, and adding more axioms does necessarily give us a lot of extra theorems.

As far as applications go, groups really are good at capturing the notion of symmetry. A concept from category theory that is relevant is the group of automorphisms of an object. What does this mean? Well, to put it roughly, an automorphism is a function which completely preserves every quality that we care about - like the symmetries of a polygon (or polyhedron) might move all the points in the ambient space, but preserve the actual polygon. We might also think about how many physical systems are translation or rotation or reflection invariant (or even time-reversal-invariance in Newtonian physics), meaning that those functions preserve all the laws of motion. And it happens that the structure of automorphisms (with respect to composition) forms a group - it is closed, since if $f$ and $g$ preserve all the desired properties, then doing $f$ then $g$ must also preserve everything desired. Moreover, every automorphism must have an inverse (and this is in fact the definition of an automorphism), as if it did not, we could conclude that some property was lost by the morphism. Associativity follows from the fact that function composition is associative, and identity follows from the fact that the identity morphism, which changes nothing, is obviously an automorphism. Outside of group theory, which studies groups generally for the sake of studying groups, this tends to be why groups arise so much - you want to know about the symmetries of an object? Well, you're looking at a group! (And, indeed, all groups can be interpreted as a set of functions closed under composition - this is Cayley's theorem)

Don't think of a group as a abstracted vector space. When you drop commutativity, you end up with objects that behave very differently from vector spaces.

The heuristic that I use is that if an operation is not commutative, then the operation should be thought of as function composition. If it is commutative, then it should be thought of multiplication, like numbers. So if we restrict to abelian groups, it turns out that they actually do behave a lot like vector spaces. Indeed, abelian groups are modules over the ring $\mathbb Z$, and vector spaces are modules over a field $k$ (usually $\mathbb R$ or $\mathbb C$). I like to think of abelian groups sort of as counting numbers, but where you can consider torsion. A commutative ring, to me, is a collection of functions, where the multiplication operation is multiplication of the outputs (think of $k[x]$, the set of all polynomials with coefficients in a field $k$, or $\mathbb Z$). A non-commutative ring is a collection of functions, where the multiplication is function composition (think $M_{n\times n}$, the collection of all $n\times n$ matricies, or the collection of all linear transformations from a vector space of dimension $n$ with basis to itself).

So if groups are like composition of morphisms (eg. functions), but the morphisms have to be invertable, then we are thinking of a collection of isomorphisms (eg. bijections). Indeed, if we look at a category (eg. all vector spaces and linear transformations between them, or groups and group homomorphisms, or sets and functions), and we look at all of the isomorphisms of an object, then that will satisfy the axioms of a group. In fact, an alternative way of describing a group is as a category with one object, and all isomorphisms are isomorphisms. To me, this is the definition of a group, because it tells you exactly what a group is for. And what is an isomorphism? It is a way of transforming an object, such that you can always reverse that transformation. A symmetry.

To cure the feeling of arbitrariness, it might be good to have examples. So we already have $Aut(X)$, the collection of automorphisms on $X$ (isomorphisms from $X$ to itself), and from that we get the symmetric group $S_n$ (automorphisms (or permutations) of a set with $n$ elements), and $GL_n(k)$, which is $Aut(V)$, where $V$ is an $n$-dimenstional $k$-vector space. Then we have useful subsets of automorphisms, like $SL_n(k)$, invertible matrices with determinant 1, and $A_n$, the collection of even permutations on a set with $n$ elements. We also have for a field extension $K/k$ (think of taking a field k and adding in some elements to get a bigger field, like take $\mathbb R$ and add in the solution to the equation $x^2 + 1 = 0$ to get $\mathbb C$) we have the automorphisms that fix the underlying field $k$, $Aut_k(K)$, which comes up in Galois theory, where you are looking at permutations of roots of polynomial equations (like complex conjugation will swap the roots of the above polynomial while keeping $\mathbb R$ fixed).

I do sympathize with you. Group theory really shouldn't be taught first in a course in abstract algebra, because all the important examples of groups come when you've seen a bunch more theory, so the whole thing comes off as fairly unmotivated. If you read Visual Group Theory by Carter, you might be able to get some motivation, especially if you look at the last part on Galois theory. Galois theory is really powerful, and solves all sorts of interesting problems that were long unsolved before the invention of Galois Theory. Problems like the impossibility of squaring a circle or of trisecting an angle with straightedge and compass, and of a formula for solving quintic equations using only radicals (as in, a quadratic equation, but for degree 5 polynomials). This last one vitally uses properties of the group $S_5$.