Why does cross product tell us about clockwise or anti-clockwise rotation?

If you use the cross product of $\vec{AB}\times \vec{AC}$ or of $\vec{AC}\times \vec{AB}$, the sign will be opposite due to the definition of the cross section. Thus you can determine in what direction you must turn around $A$ to reach $C$ from $B$ by looking at the sign of the cross product.

In terms of angles if $\vec{AB}$ and $\vec{AC}$ are in the $xy$ plane : $$\vec{AB}\times \vec{AC} = (|AB||AC|\sin\theta) \hat{z}$$ $$\vec{AC}\times \vec{AB} = (|AB||AC|\sin(-\theta))\hat{z} = -(|AB||AC|\sin(\theta))\hat{z}$$ Thus the angle becomes negative when you switch the direction - it's a bit like saying that to get from 12 o'clock to 3 o'clock you need to go $90^\circ$, but to go from 3 o'clock to 12 o'clock you need $270^\circ = 270^\circ - 360^\circ = -90^\circ$.


Given three non-coplanar(!) vectors $v_1, v_2, v_3$ in 3d space, these can be smoothly transformed into another triplet of vectors $w_1, w_2, w_3$ without becoming coplanar at any intermediate moment - or not. Interestingly, if you can't obtain $w_1,w_2,w_3$, then you can obtain $w_1, w_2, -w_3$ and also $w_2,w_1,w_3$.

One can find out beforehand for given $v_1,v_2,v_3$ and $w_1, w_2, w_3$ whether or not such a smooth transition is possible: Check if $v_1\times v_2$ and $v_3$ are on the same side of the plane spanned by $v_1, v_2$. If so, we say the three vectors are positively oriented, otherwise not. Do the same check with the $w_i$. The smooth transition described is possible if and only if the orientation is the same. Thus for two non-collinear vectors in 3d space, we can always find a "standard" third vector to make a positively oriented triple: $v_1\times v_2$.

Considering a nondegenerate triangle $ABC$ in a plane, we have a similar situation: It can be smoothly transformed to $DEF$ without becoming degeneate at an intermediate time iff $\vec{AB}\times \vec{AC}$ and $\vec{DE}\times \vec{DF}$ are in the same halfspace with respect to the plane.


The geometric interpretation of the cross product $\vec{a} \times \vec{b}$ is that it gives us a vector that is perpendicular to both $\vec{a}$ and $\vec{b}$, and has length $\left \| \vec{a} \right \| \left\| \vec{b} \right \| \sin \theta$. Howwever, there are 2 such vectors (which point in the exact opposite direction), so this isn't sufficient. We still require the 'right hand rule' to tell us which specified direction the vector points in.

Specifically, if the points $A, B$ are anti clockwise about the origin, then the vector $\vec{a} \times \vec{b}$ will point out of the page, while the vector $\vec{b} \times \vec{a}$ will point into the page. We can tell which direction the vector is pointing at, but taking the dot product with $(0, 0, 1)$. If $( \vec{a} \times \vec{b} ) \cdot (0, 0, 1) >0$, then the points are anti clockwise. If $( \vec{a} \times \vec{b} ) \cdot (0, 0, 1) <0$, then the points are clockwise. If equality holds, then the vectors are parallel to each other.


If you rather look at the algebraic interpretation of the cross product, then $( a_1, a_2 , a_3 ) \times (b_1, b_2, b_3) = \det \begin{pmatrix} a_1 & a_2 & a_3 \\ b_1 & b_2 & b_3 \\ \vec{i} & \vec{j} & \vec{k} \\ \end{pmatrix}$. In the plane, we have $a_3 =0, b_3 = 0$.

Recall that if points $O, A, B$ are in anti clockwise order, then triangle $OAB$ has positive area equal to $\frac {1}{2} \left | \begin{matrix} 0 & a_1 & b_1 & 0 \\ 0 & a_2 & b_2 & 0 \\ \end{matrix} \right | = \frac {1}{2} (a_1 b_2 - a_2 b_1 )$. As such, the orientation can be described by looking at just the signage of this term, which happens to correspond with the coefficient of vector $\vec{k}$ in $\vec{a} \times \vec{b}$. We can recover this term (and hence it's sign) by looking at $ (\vec{a} \times \vec{b} )\cdot \vec{k}$