Automorphism groups and symmetric groups

Solution 1:

Automorphism of a set is an arbitrary permutation of its elements. An automorphism of a group is permutation of its elements which preserves the operation, i.e. $\varphi(xy)=\varphi(x)\varphi(y)$. Since every group $G$ is a set, you can look at two possible automorphism groups: one - $\operatorname{Aut}_{Set}(G)$ as a set and the other $\operatorname{Aut}_{Gp}(G)$ as a group. Cleraly $\operatorname{Aut}_{Gp}(G)\leq \operatorname{Aut}_{Set}(G)$, but usually they are not equal.
When talking about groups, the notion $\operatorname{Aut}(G)$ means the set of group-automorphism of $G$.
In your case, $\operatorname{Aut}(S_n)$ denotes the group-automorphisms of $S_n$, so there is no contradiction with the previous statements. BTW, there is no problem with $n=1$, since $S_1=\{id\}$ and $\operatorname{Aut}(S_1)=\{id\}$. To illustrate the problem in $S_2=\{id,(1,2)\}$, observe that there are two automorphisms of $S_2$ as a set: $$\begin{array}{c}id\mapsto id\\ (1,2)\mapsto(1,2)\end{array} \hspace{10pt} {\rm{and}}\hspace{10pt}\begin{align*}id &\mapsto (1,2)\\ (1,2)&\mapsto id\end{align*}$$ but the right automorphism is not a group-automorphism, hence there exists only one group-automorphism of $S_2$.

Solution 2:

In general, an automorphism of an object $A$ is a morphism $f\colon A\to A$ such that there exists a morphism $g\colon A\to A$ such that the composositions $f\circ g$ and $g\circ f$ are both the identity morphism of $A$.

To step from this abstract concept to some concrete situation, one must specify what kind of objects one talks about and what a morphism is. For sets (or as one says: in the category Set), an object is, of course, just a set and a morphism is simply a map. Then an automorphism of a set $A$ is simply an arbitrary bijective map $A\to A$.

On the other hand, for groups (in the category Grp), the objects are groups and the morphisms are group homomorphisms, i.e. maps that "respect the group law". For example, an automorphism of a group necessarily maps the identity element to itself, whereas a bijection (or "set-automorphism") need not obey this restriction.