Must large (infinite) groups have large automorphism groups?

Starting from this previous question about infinite groups with finite automorphism groups, I quickly landed in the following paper:

J.T. Hallet and K.A. Hirsch, Torsion-free groups having finite automorphism groups I, J. Algebra 2 (1965) 287-298.

In the introduction, they say:

Preliminary results by de Groot [7], Hulanicki [10], Fuchs [5], and Saqiada [11] showed successively that for every cardinal number $r$ less than $2^{\aleph_0}$, $2^{2^{\aleph_0}}$, $2^{2^{2^{\aleph_0}}}$ there are torsion-free abelian groups $G$ of rank $r$ with $|\mathrm{Aut} (G)| = 2$, and in 1959 Fuchs [6] stated that there is no restriction whatever on the cardinal number $r$ of the rank of such a group. True, a flaw in Fuchs’ argument was revealed by Corner [3], but he at least was able to save the result for all ranks $r$ smaller than the hypothetical first “strongly inaccessible” cardinal number.

The interest of the article is to find which finite groups occur as automorphism groups of infinite groups, so they do not appear to go further into this. But note that since the existence of strongly inaccessible cardinals is unprovable in ZFC, I expect you would need to go to model theory and play with variants of large cardinal axioms in order to be able to get an affirmative answer to your question, even for $\kappa=2$.

ADDED: Jeremy Rickard has pointed out in comments that Saharon Shelah removed the restriction on $\lambda$ being less than the first strongly inaccessible cardinals in 1974, nine years after the paper I am quoting. So the answer to the question is an inequivocal “no”, since it fails for $\kappa=2$.