What does it mean when people say that groups are a study of symmetry?
I see many people make remarks to the effect that groups have basic symmetry properties. I am familiar with Cayley's Theorem and the symmetric groups $S_N$. However, when I think of the symmetric group, I think of a group of permutations, not anything to do with symmetries. Additionally, I have been told that semi-groups lack the basic symmetry properties that groups have. (Note: I only know the basic definition of a semi-group.)
What does symmetry mean in this context?
It's good that you keep symmetry groups and symmetric groups well separate, they each use symmetry in a slightly different sort of way!
Classically, things like parabolas were considered symmetric: They have an axis of symmetry and reflection across that axis leaves the "footprint" of the parabola unchanged (even though individual points may get shuffled around). Regular triangles, squares (regular polygons in general), circles, and tilings (like a floor tile, or brick patterns) are all two-dimensional objects that have a lot (OK, usually at least a "nontrivial amount") of symmetry: they "look the same" from several different viewpoints.
Eventually, mathematicians formalized a symmetry to mean an isometry (function that preserves the distance between any two points) of Euclidean space that again leaves the "footprint" of (set of all points comprising) the object unchanged, while potentially shuffling the individual pieces. This is what Bye_world is referencing in the comments, and is really a pretty radical way to think about things!
The full set of symmetries of an object do form a group: The composition of symmetries (which, remember, are really just special functions from $\Bbb R^n$ [or related spaces] to itself) is yet another symmetry. Function composition is associative. The identity map is the identity symmetry, and it doesn't do anything. Finally we can "undo" any symmetry; for example performing a counterclockwise rotation of $28^\circ$ to undo a clockwise rotation of $28^\circ$, or translating in the opposite direction to undo a translation, etc.
This is why semigroups aren't really a very good language to talk about symmetry.
We don't have an identity. We can't be guaranteed that we can just "do nothing" to the object in consideration. Not only is this not-so-good (TM) because doing nothing is nice, but also because
We aren't guaranteed inverses -- let alone the ability to define them, if we can't be sure we don't have an identity element! This really doesn't get along well with our picture of symmetries of an object: Two viewpoints of an object aren't really "the same" if we can't switch freely between them, but only from one to the other, and not back: We need inverses, and we need an identity to talk about inverses.
Sidenote: More generally, we have automorphisms of mathematical objects. These are maps from an object to itself that preserves some essential structure; a more abstract version of symmetry. You've probably encountered group automorphisms, or graph automorphisms. I personally tend to think in the language of "symmetries of a graph" rather than "automorphisms of a graph" unless I need to be fancy for some reason. This is distinct from a symmetry in the "isometry" sense, although the line gets a little blurry, namely because we like to draw ( = "realize") graphs in $\Bbb R^2$, and then the notions sometimes coincide!
Again, automorphisms come together to naturally form a group, and "automorphism" is a term that can be applied to a surprisingly large number of kinds of structures. This is the sense in which groups are the language of symmetry: If you have things that "act like" (or are) symmetries, then they form a group!
Polynomials are another place where "symmetric" is somewhat commonly used: There are things called symmetric polynomials (or more generally, symmetric functions), like $f(x, y) = x^2y + xy^2$ whose values don't change when we permute the variables; so $f(x,y) = f(y, x)$ above. I forget any surprising places these show up, but I know that Newton did some work with them, predating Galois and the formal definition of a group.
At any rate, I had suspected this is where the term "Symmetric Group" got its name (since the permutations of the variables do indeed form symmetric groups), and this answer confirms it, by quoting a MathOverflow post quoting the pioneer Burnside!
All the Symmetries of an object $F$ in a space $X$ forms a group.
Here $X$ is a non-empty set, with or without some structure, and $F$ is a subset (call figure) in $X$; then symmetry of $F$ in $X$ means a bijective map from $X$ to $X$, preserving structure of $X$, and taking $F$ to itself.
Initially, it would be better to see real life examples: take rectangle in plane; its group of symmetries is Klein-4 group. Other examples appear in comments.
Next, consider the following symmetric figure:
A symmetry of this object is a movement (arrangement) of sticks, which take the figure to itself. There are 16 sticks, any permutation of these sticks takes the figure to itself, hence the groups of symmetries of this figure is $S_{16}$.
You already said that you are familiar with Cayley's theorem. We can say that every group occurs as symmetries of some object: any group $G$ can be embedded in some symmetric group, and symmetric group is group of symmetris of some object, like above picture. Q.E.D.
Summarizing: symmetries of any object can be described by a group and behavior of symmetries is expressed by the action of group (on the object). The abstract study of groups is either study of group or its action. Both, the group and group action are coming from symmetries of some object; hence the study of groups is usually called the study of symmetries.