Quasi-inverse of an equivalence of categories is unique up to unique isomorphism

It's simply not true: the isomorphism is not unique. Indeed, you could have $G=G'$, in which case the claim is that any functor which is an equivalence has no automorphisms besides the identity. This is obviously false. For instance, the identity functor $1:Ab\to Ab$ has multiplication by $-1$ as a nontrivial automorphism. Or if $A$ is any group considered as a one-object category, any element of the center of $A$ gives an automorphism of the identity functor.

You might demand for the isomorphism to be compatible with the natural transformations $\alpha$ and $\beta$, but then it need not even exist. Indeed, such a compatible isomorphism would preserve whether the $\alpha^{-1}$ and $\beta$ are the unit and counit of an adjunction between the two functors. This is true for some choices of $(G,\alpha,\beta)$ but not others. Again, you can get an easy counterexample using groups as one-object categories. Let $A$ be a group and $F=G=G'$ be the identity functor, let $\alpha$, $\alpha'$, and $\beta$ be the identity, but let $\beta'$ be some nontrivial central element of $A$.

The right way to get a unique isomorphism is to only consider adjoint equivalences, that is equivalences $(F,G,\alpha,\beta)$ for which $\alpha^{-1}$ and $\beta$ are the unit and counit of an adjunction. For any quasi-inverse $G$ of $F$, there exists a choice of $\alpha$ and $\beta$ which do form an adjoint equivalence. Then, between any two such choices of $(G,\alpha,\beta)$ there is a unique isomorphism which is compatible with the $\alpha$ and $\beta$ maps. This is just a corollary of the more general statement that the left adjoint of a functor is unique up to unique isomorphism preserving the adjunction, which is an immediate consequence of Yoneda's lemma when you view the adjunction in terms of Hom-sets. (In terms of your attempted argument, the condition $G(\alpha_-) \circ \beta^{-1}_{G(-)}=id_{G(-)}$ which you want is exactly one of the zigzag equations that the unit and counit of an adjunction are required to satisfy.)