What properties must have two orthonormal matrix to commute?

Two normal matrices commute if and only if they are diagonalizable with respect to the same orthonormal basis. This is also equivalent to being unitarily equivalent, that is $B=UAU^*$ for some unitary $U$.

Here is a sketch of the argument. Suppose that $A,B$ are normal and that they commute.

Note: a matrix $A$ is "normal" if it commutes with its adjoint $A^*$ (conjugate transpose). For real matrices, $A$ normal means simply that $AA^t=A^tA$. A matrix $U$ is "unitary" if $U^*U=UU^*=I$; when $U$ is real, unitary is the same a orthogonal.

• Any normal matrix is of the form $UDU^*$, for a unitary $U$ and diagonal $D$. This follows from Schur's Triangulation Theorem.

• As mentioned in the aforementioned article, Schur's decomposition can be applied simultaneously (i.e., with the same unitary) to commuting matrices.

• By the above, since $AB=BA$, we have $A=UDU^*$, $B=UEU^*$ with $D,E$ diagonal.

Conversely, if $A,B$ are diagonalizable with respect to the same orthonormal basis, we have $A=UDU^*$, $B=UEU^*$, where $D,E$ are diagonal and $U$ is the unitary doing the change of basis from said orthonormal basis to the canonical one. Thus $$ AB=UDU^*UEU^*=UDEU^*=UEDU^*=UEU^*UDU^*=BA. $$