How group action works.

$\newcommand{\Map}{\textrm{Map}}\newcommand{\Sym}{\textrm{Sym}}$Unfortunately, you are interpreting something wrong… Let's try to make more sense out of it.

What that says to me is it is a homomorphism (function) $f\colon X\to S$ where $S$ is a set of bijective functions on $X$.

No, it doesn't say that. There are three different objects here, but you keep confusing them with other (in different ways throughout your post). Those three objects are:

• a group $G$,
• a set $X$,
• and the symmetric group of all bijective functions on $X$, which we may denote $S=\Sym(X)$.

Now, the definition of a group action (of a group $G$ on a set $X$) says that is it a group homomorphism $f\colon G\to S$, i.e. $f\colon G\to\Sym(X)$. Note that the domain of the group action $f$ is $G$, not $X$ as you said.

… if we have … a group $(∗,\{4,5,6\})$ …

This is a bit hard to understand. I guess you're saying that we have a group $G=(∗,\{4,5,6\})$. Okay, so $\{4,5,6\}$ is the underlying set of this group, but it's not quite clear how the operation $*$ works. Since this is not one of the common standard notations for a concrete group (like $\mathbb{Z}$ or $V_4$), you should define it. Of course, since you're having a general discussion here and don't use this operation explicitly, I can't say that there's anything wrong with it per se. It is possible to define a group operation $*$ on the set $\{4,5,6\}$. But it would be a bit unusual considering the choice of the symbols for the elements of the group (for example, one of them — either $4$ or $5$ or $6$ — will have to be the identity element). So the only reason I am pointing this out is because I suspect it is also a sign of one of your confusions.

… the symmetric group of $X$ would be something like $S=\{1\to2,2\to3,\ldots\}$ …

Sorry, but this doesn't make much sense. We can say that an individual element of the symmetric group looks more or less like that. For example, if $X=\{1,2,3\}$, then one of the elements of $S=\Sym(X)$ is the bijective mapping $\sigma_1=\{1\mapsto2,2\mapsto3,3\mapsto1\}$, or rather $\sigma_1=\{(1,2),(2,3),(3,1)\}$ in ordered pairs notation. Another examples of an elements of $S=\Sym(X)$ is the bijective mapping $\sigma_2=\{1\mapsto2,2\mapsto1,3\mapsto3\}=\{(1,2),(2,1),(3,3)\}$. But then $S=\Sym(X)$ is the set of all such bijective mappings. In this example (hold your breath): $$\begin{align} S&=\{\sigma_1,\sigma_2,\sigma_3,\sigma_4,\sigma_5,\sigma_6\}, \quad \text{where}\\ \sigma_1&=\{1\mapsto2,2\mapsto3,3\mapsto1\}=\{(1,2),(2,3),(3,1)\},\\ \sigma_2&=\{1\mapsto2,2\mapsto1,3\mapsto3\}=\{(1,2),(2,1),(3,3)\},\\ \sigma_3&=\{1\mapsto3,2\mapsto2,3\mapsto1\}=\{(1,3),(2,2),(3,1)\},\\ \sigma_4&=\{1\mapsto1,2\mapsto3,3\mapsto2\}=\{(1,1),(2,3),(3,2)\},\\ \sigma_5&=\{1\mapsto3,2\mapsto1,3\mapsto2\}=\{(1,3),(2,1),(3,2)\},\\ \sigma_6&=\{1\mapsto1,2\mapsto2,3\mapsto3\}=\{(1,1),(2,2),(3,3)\}, \end{align}$$ listed in no particular order.

… and $\varphi$ would be something like $f\colon\{4,5,6\}\to S$.

Yes, under three conditions:

• you understand correctly what $S$ is;
• you clearly define a group structure, i.e. the multiplication operation $*$ on the group $G=\{4,5,6\}$;
• and you make sure that $f\colon G\to S$ is group homomorphism between the group $G=\{4,5,6\}$ with the operation $*$ and the group $S=\Sym(X)$ with composition as the group operation.

Let $\newcommand{\Map}{\textrm{Map}}\newcommand{\Sym}{\textrm{Sym}}$ us look first at sets $A$, $B$ and $C$. Write $\Map(A,B)$ for the set of all mappings (functions) from $A$ to $B$. There is a natural correspondence (in the sense of category theory) $$\Map(A\times B,C)\leftrightarrow\Map(A,\Map(B,C)).$$ Computer scientists call this currying after Haskell Curry. It says that maps $f:A\times B\to C$ correspond to maps $F:A\to\Map(B,C)$. How does this work? If $f:A\times B\to C$ and $a\in A$, define $F_a:B\to C$ by $F_a(b)=f(a,b)$. Then $F_a\in\Map(B,C)$ and $F:a\mapsto F_a$ is a map from $A$ to $\Map(B,C)$.

Now let's get to group actions. If $G$ is a group and $X$ a set, then a group action is an element of $\Map(G\times X,X)$ or equivalently $\Map(G,\Map(X,X))$ satisfying certain conditions. If we consider it as a map $F:G\to\Map(X,X)$ then its image must lie in $\Sym(X)\subseteq\Map(X,X)$, the collection of bijections from $X$ to $X$. Moreover $F$ must be a group homomorphism. Then $F$ corresponds to $f:G\times X\to X$ by currying. The condition that $F$ be a group homomorphism translates to $f(e,x)=x$ and $f(g_1g_2,x)=f(g_1,f(g_2,x))$. If we write $f$ in infix notation as $f(g,x)=g\cdot x$ then these become $e\cdot x=x$ and $(g_1g_2) \cdot x)=g_1\cdot(g_2\cdot x)$.