Cross product: matrix transformation identity

Recall that $u\times v$ is the only vector $w\in\mathbb{R}^3$ that satisfies $$\forall\, x\in\mathbb{R}^3,\quad \det(u,v,x)=\langle w,x\rangle$$ So, we have $$\eqalign{\det(M)\langle a\times b,x\rangle&= \det(M)\det(a,b,x)\cr&=\det(Ma,Mb,Mx)\cr &=\langle Ma\times Mb,Mx\rangle\cr &=\langle M^T(Ma\times Mb),x\rangle\cr } $$ Thus $$\forall\, x\in\mathbb{R}^3,\quad \langle\det(M)( a\times b),x\rangle=\langle M^T(Ma\times Mb),x\rangle$$ It follows that $\det(M)( a\times b)=M^T(Ma\times Mb)$ or equivalently, for an invertible matrix $M$, $$\det(M)(M^T)^{-1}( a\times b)= Ma\times Mb $$

Remark. The most general formula is $${\rm Co}(M)( a\times b)= Ma\times Mb$$ where ${\rm Co}(M)$ is the matrix of cofactors of $M$. Here we do not need $M$ to be invertible. This follows from the proved formula by continuity.

Building on Jyrki's comment, the relationship between the wedge and cross products is of some interest here.

This is most easily formulated using clifford algebra, instead of just exterior algebra. In clifford algebra, the cross product and the wedge product of vectors are related through duality:

$$a \times b = -i (a \wedge b)$$

Multiplication by the pseudoscalar $i$ performs the duality operation.

The extension of linear operators, which are represented by matrices, to multivectors is simplest when viewed in terms of the wedge product:

$$M(a \wedge b) \equiv M(a) \wedge M(b)$$

This extends to any number of wedges, so Omran's proof using clifford algebra would look something like this:

$$(\det M)(a \wedge b \wedge x) = M(a \wedge b \wedge x) = [M(a) \wedge M(b)] \wedge M(x) = -i [M(a) \times M(b)] \cdot M(x) = \ldots$$

A more direct approach, albeit more difficult to prove first principles, is to use to following inverse formula:

$$[M^T]^{-1}(a) = [M(i)]^{-1} M(i a) $$

Given that $i (a \times b) = a \wedge b$, we get the result

$$[M^T]^{-1}(a \times b) = (\det M)^{-1} i^{-1} M(a \wedge b) = (\det M)^{-1} M(a) \times M(b)$$