Characterization of nonabelian group $G$ such that for all $x,y\in G$, $xy\neq yx\implies x^2=y^2$.

Let $G$ be a nonabelian group such that for all $x,y\in G$, $xy\neq yx\implies x^2=y^2$. (Or equivalently, for all $x,y\in G$, either $xy=yx$ or $x^2=y^2$.)

Let $g\in G\setminus Z(G)$. Then $gh\neq hg$ for some $h\in G$. Therefore $gh^{-1}\neq h^{-1}g$ and $g^{-1}h\neq hg^{-1}$. By the assumption, we get $g^2=g^{-2}=h^2=h^{-2}$. This implies that $g^4=1$.

Next let $z\in Z(G)$. Since $(zg^{-1})h\neq h(zg^{-1})$, it follows that $(zg^{-1})^2=h^2$, which implies that $z^2=g^2h^2=1$.

In short, $G$ is a $2$-group of exponent $4$. Such a group exists as the quaternion group $Q_8$ fulfils the property. Are there any characterization on groups with this property? Or are there any other group that fulfils the property?

We will prove that any such group is the direct product of a quaternion group $Q_8$ and an elementary abelian $2$-group.

As you have shown, $G$ has exponent $4$ (as it cannot have exponent $2$, since such groups are elementary abelian).

Let $g\in G\setminus Z(G)$ have order $2$, and let $h\in G\setminus C_G(g)$. Then $g^2=1$, so $h^2=1$ as well. But $\langle g,h\rangle$ is generated by two involutions and is non-abelian, so must be $D_8$. However, then $gh$ has order $4$ and does not commute with $g$ either, a contradiction.

Thus all involutions are central. Let $g$ and $h$ be non-commuting elements of order $4$, and let $H$ be generated by them. If $x\in C_G(H)$ then $g$ and $hx$ cannot commute either, whence $g^2=(xh)^2=x^2h^2$. Since $g^2=h^2$, we see $x^2=1$, and $C_G(H)$ (and therefore also $Z(G)$) consists of exactly the involutions in $G$.

Notice that $g^4=1$, so $g^{-1}=zg$, where $z=g^2\in Z(G)$. If $[g,h]\neq 1$ then $g^2=h^2=z$, and again $h^{-1}=zh$. Also if $[g,h]\neq 1$ then $[g,gh]\neq 1$, so $(gh)^2=z$. Thus $$h^{-1}gh=h^3gh=hghz=hghgg=zg=g^{-1}.$$ In particular this means that every cyclic subgroup is normal, since $g^h=g^{\pm 1}$ for all $g,h\in G$.

By assumption the two cyclic subgroups do not commute, thus they generate a $Q_8$ subgroup $H$ (noting that $\langle g\rangle\cap \langle h\rangle=\langle g^2\rangle$ as above). Also, if $X=\langle g\rangle$, then $N_G(X)/C_G(X)$ has order $2$, so $|G:C_G(X)|=2$. By symmetry, the same holds for $Y=\langle h\rangle$.

We aim to show that $G=HZ(G)$. But $C_G(H)=C_G(X)\cap C_G(Y)$ has index at most $4$, whereas $H/Z(H)$ has order $4$, so $G=HC_G(H)$. Since $C_G(H)$ is elementary abelian, $C_G(H)=Z(G)$, so $G=HZ(G)$.

Thus $G=HZ(G)$, $Z(G)$ is elementary abelian, and so $G\cong H\times Z$ for some elementary abelian group $Z$.

The direct product $G$ of the quaternion group and the group $C$ of order two has the property. Let $a$ be the order-two element of the quaternion group, let $b$ be the non-identity element of $C$. The center of $G$ is $Z=\{\,(1,1),(1,b),(a,1),(a,b)\,\}$; the elements of $G$ of order two are the non-identity elements of $Z$; all the elements of $G$ other than those in the center have order four; each element of order four has square $(a,1)$.

So, if two elements of $G$ don't commute, then they are both of order four, and they have the same square.