Are these exactly the abelian groups?

Suppose $G$ satisfies your condition. Let $x,y \in G$ be distinct elements of $G$ different from $1$ so that the listed elements of all sets in this answer are distinct. Pick $g=x$ and $A=\{1,x,y\}$ Then $Ag = hA$ implies

$$ \{ x, xx, yx \} = \{ h, hx, hy \} $$

Case 1: $x=h$

Then $\{ xx,yx \} = \{ xx,xy \}$, and $yx = xy$

Case 2: $x = hx$

Then $h=1$ and $\{ xx, yx \} = \{ 1, y \}$.

If $yx = y$, then $x=1$ and $xy=yx$. Otherwise $yx=1$, and so $y=x'$ and $xy=1=yx$.

Case 3: $x=hy$

Then $h=xy'$ and $\{ xx, yx \} = \{xy',xy'x\}$

If $xx = xy'$, then $x=y'$ and thus $xy=1=yx$. Otherwise $xx = xy'x$ and thus $1=y'$, and again $xy=yx$.

In all cases where $x,y,1$ are distinct, we've shown $x$ and $y$ commute. Thus $G$ is an abelian group.

Assume $ab\ne ba$. Let $A=\{1,a\}$, $g=b$. Then there is $h\in G$ such that $\{h,ha\}=\{b,ab\}$. This needs $h=b\lor h=ab$. In the first case $ha=ba\ne ab$, so this fails. Therefore $h=ab$ and $ha=aba=b$. Similarly, $bab=a$. This implies $aa=abab=bb$. We conclude $$a=bab=bbabb=aaaaa, $$ hence $a^4=1$ and similarly $b^4=1$.

Now take $A=\{1,a,b\}$ and $g=b$. Then there is $h\in G$ such that $\{h,ha,hb\}=\{b,ab,b^2\}$.

• $h=b$: Then $hb=b^2$ implies $ba=ha=ab$, contradiction
• $h=ab$: Then $ha=aba=b$ implies $hb=b^2$, i.e. $a=1$ and of course $ab=ba$, contradiciton.
• $h=b^2=a^2$: Then $ha=a^3=a^{-1}\ne b$, hence $ha=ab$, i.e. $a^2=b$ and of course $ab=ba$, contradiction