How many group actions are there of $Q_8$ on a 4 element set.

Solution 1:

Your idea to count homomorphisms $Q_8\to S_4$ is a good one, with the caveat that we only want to count them up to conjugacy in $S_4$.

Suppose $h\colon Q_8\to S_4$ is a homomorphism. Then $\ker(h)$ is a normal subgroup of $Q_8$, so it is one of: $\{1\}$, $\{1,-1\}$, $\{1,i,-1,-i\}$, $\{1,j,-1,-j\}$, $\{1,k,-1,-k\}$, or $Q_8$.

If $\ker(h) = \{1\}$, then $h$ is injective. Since $S_4$ has no subgroup isomorphic to $Q_8$, this case is impossible.
If $\ker(h) = \{1,-1\}$, then $\mathrm{im}(h)\cong Q_8/\{1,-1\}\cong K_4$. Now there are $4$ subgroups of $S_4$ isomorphic to $K_4$. Three of them are generated by two disjoint transpositions: $\{\varepsilon, (a\,b),(c\,d),(a\,b)(c\,d)\}$. The fourth consists of the identity and all products of two disjoint transpositions: $\{\varepsilon,(1\,2)(3\,4),(1\,3)(2\,4),(1\,4)(2\,3)\}$. Once we've pinned down the image, there are $6$ homomorphisms with that image, corresponding to the $6$ automorphisms of $K_4$ (each non-identity element of $Q/\{1,-1\}$ can be mapped to any element of the image). So in total there are $24$ homomorphisms with kernel $\{1,-1\}$. But we only want to count actions up to isomorphism. Homomorphisms with image of the form $\{\varepsilon, (a\,b),(c\,d),(a\,b)(c\,d)\}$ correspond two actions in which two of $\pm i,\pm j,\pm k$ act as disjoint transpositions and the third acts as the product of the two disjoint transpositions. Up to isomorphism there are $3$ of these, determined by which of $\pm i,\pm j,\pm k$ acts as a product of two disjoint tranpositions. Homomorphisms with image $\{\varepsilon,(1\,2)(3\,4),(1\,3)(2\,4),(1\,4)(2\,3)\}$ correspond to just one action up to isomorphism, in which each non-identity element acts as a product of disjoint transpositions. So there are $4$ actions with kernel $\{1,-1\}$, up to isomorphism.
If $\ker(h) = \{1,i,-1,-i\}$, then $\mathrm{im}(h)\cong Q_8/\{1,i,-1,-1\}\cong C_2$. Now a homomorphism $C_2\to S_4$ is determined by an element of $S_4$ of order $2$. There are $9$ such elements: $6$ transpositions $(a\,b)$, and $3$ elements which are products of two disjoint transpositions $(a\,b)(c\,d)$. In each case, all homomorphisms give isomorphic actions, but the two cases correspond to non-isomorphic actions. So we have $9$ homomorphisms and $2$ actions up to isomorphism in this case.
The same analysis works if $\ker(h) = \{1,j,-1,-j\}$ or $\{1,k,-1,-k\}$. So there are $9$ more homomorphisms and $2$ more actions up to isomorphism in each of these cases.
If $\ker(h) = Q_8$, $h$ is the trivial homomorphism. This corresponds to the trivial action.