Why does $\det (A)$ change sign when any $2$ columns of $A$ are interchanged?
Yay, that is my favourite definition of the determinant haha.
Your reasoning is allright, but you could be more precise. Suppose for example that you swapped columns $1$ and $2$. Then you can show that $A_1x_1=Ax_2$, that $A_1x_2=Ax_1$ and $A_1x_i=Ax_i$ for each $i\neq 1,2$. Similarly picking any two adjacent columns $j,j+1$.