Is there a non-abelian group $G$ such that $\operatorname{Aut}(G)$ is abelian?

Solution 1:

A non-abelian group can have an abelian group of isomorphisms (automorphisms). An affirmative answer to this question was given by G. A. Miller in 1913 . Title of the paper is "A non abelian group whose group of isomorphism is abelian" He constructed a non-abelian group $G$ of order $64$ such that $Aut(G)$ is abelian and has order $128$.

Added In this paper Author has defined $p + 1$ non-isomorphic groups of order $p^8$ (for each $p$) whose automorphism groups are elementary abelian of order $p^{16}$.

This link may also help you.