Can a group have a subset that is stable under all automorphisms, but not under inverse?

The title really says it all. For any group $G$ the map $g\mapsto g^{-1}$ gives a (canonical) isomorphism from $G$ to the opposite group $G^\mathrm{op}$; this marks a difference with for instance non-commutative rings where the two might not be isomorphic. As a consequence there is for instance no fundamental difference between studying left and right group actions, even for a fixed group$~G$.

So while groups cannot be in any manner "naturally left-handed", whatever that might be construed to mean, there is still the potential possibility of being able to point to some "chiral" subset of the group, a subset that might have some relation to the group without being in the same relation to the opposite group. It is quite evident that notions such as normaliser or centraliser applied to some (set of) elements will never yield such subset; indeed anything that is a subgroup will not do because it is closed under inverse.

Which brought me to my question:

Can a group $G$ have a subset closed under the action of $\operatorname{Aut}(G)$, but not under the map $g\mapsto g^{-1}$?

Since one has in particular the inner automorphisms, one need only consider unions of conjugacy classes.

I can see no reason why this should be impossible, but it is also hard to construct an example.

• For Abelian groups $g\mapsto g^{-1}$ is an automorphism, so no luck here
• On the other hand most symmetric groups have trivial outer automorphism groups; however here every $g$ is conjugate to $g^{-1}$, no luck either
• In alternating groups a conjugacy class of $S_n$ might fall apart, an an element might not be conjugate to its inverse; the two conjugacy classes would still however be related by an outer automorphism (coming from $S_n$)
• In $GL(n,K)$ most elements are not conjugate to their inverse; however there is an obvious out automorphism of transpose-inverse, and every invertible matrix is similar to its transpose (due to the theory of f.g. modules over a PID, leading to classification of similarity by invariant factors).
• Most "natural" subgroups of $GL(n,K)$ would suffer from the same "outer automorphism" phenomenon as alternating groups do, so they do not look promising.

That's where I am now; I unfortunately do not know a lot of examples where I know enough to easily see whether I can find such subsets. Maybe a good candidate is the general linear group over a skew field (division ring), since the result of similarity to one's transpose does not hold there, as far as I know. But I don't know enough about automorphisms of these groups to known whether they are good candidates.

There are many examples in the literature of finite $p$-groups of nilpotence class $2$ in which all automorphisms are central, that is, for every automorphism $\alpha$ and every $x \in G$, one has $\alpha(x) = x z$ for some $z \in Z(G)$.

If $G$ is such a group, with $p > 2$, and $g \in G \setminus Z(G)$, then there is no automorphism taking $g$ to $g^{-1}$, otherwise $g^{-1} = g z$ for some $z \in Z(G)$, and $g^{2} \in Z(G)$, which forces $g \in Z(G)$ as $p > 2$.

So you may take the orbit of any such $g$ under the automorphisms of $G$ as your subset.

As a reference, you may take this paper.