Finding rotation of 3 given lines in 3D until intersection with 3 other given lines

Solution 1:

Let us change somehow you notation and define, for the first set of lines $$ \begin{array}{l} {\bf P}_{\,{\bf 1}} = \left( {\begin{array}{*{20}c} {p_{\,x,\,1} } & {p_{\,x,\,2} } & {p_{\,x,\,3} } \\ {p_{\,y,\,1} } & {p_{\,y,\,2} } & {p_{\,y,\,3} } \\ {p_{\,z,\,1} } & {p_{\,z,\,2} } & {p_{\,z,\,3} } \\ \end{array}} \right)\quad \quad {\bf V}_{\,{\bf 1}} = \left( {\begin{array}{*{20}c} {v_{\,x,\,1} } & {v_{\,x,\,2} } & {v_{\,x,\,3} } \\ {v_{\,y,\,1} } & {v_{\,y,\,2} } & {v_{\,y,\,3} } \\ {v_{\,z,\,1} } & {v_{\,z,\,2} } & {v_{\,z,\,3} } \\ \end{array}} \right) \\ {\bf \Lambda }_{\,{\bf 1}} = \left( {\begin{array}{*{20}c} {\lambda _{\,1,1} } & 0 & 0 \\ 0 & {\lambda _{\,1,2} } & 0 \\ 0 & 0 & {\lambda _{\,1,3} } \\ \end{array}} \right) \\ \end{array} $$ while using the index $2$ for the second set.

Given the values of each parameter $\lambda_{1,k}$, we are defining one point on each of the three lines of the first set, which constitutes a triangle in 3D.
Same for the second set of lines. $$ {\bf T}_{\,{\bf 1}} = {\bf P}_{\,{\bf 1}} + {\bf V}_{\,{\bf 1}} \,{\bf \Lambda }_{\,{\bf 1}} \quad {\bf T}_{\,{\bf 2}} = {\bf P}_{\,{\bf 2}} + {\bf V}_{\,{\bf 2}} \,{\bf \Lambda }_{\,\,{\bf 2}} $$

If I interpretated your question correctly, we are to
find the values of $\lambda_{1,k}$ and $\lambda_{2,k}$ such that the two triangles defined in each set might be made to coincide upon a rigid rotation around the origin, and find the parameters of the rotation realizing that.

Since you wrote that
line $i$ in set 1 shall intersecate line $i$ in set 2
then the triangles shall also coincide
vertex $i$ to vertex $i$.

That means that the pyramid $OT_2$ shall rigidly rotate around the vertex in $O$ and come to coincide with the pyramid $OT_1$.
And being the rotation rigid, then all the dimensions of the pyramid (lengths, areas and volume) shall be preserved.

We can translate the above into the following conditions

• the distance from the origin of each vertex of $T_1$ shall equal the distance of the corresponding vertex of $T_2$, and
• the angle between each couple of edges from the origin to two vertices of $T_1$ shall equal the corresponding on $T_2$, and
• (to avoid reflection) the signed volume of the pyramid $OT_1$ shall equal that of $OT_2$, when the vertices are listed in the same order.

That means that it should be (indicating the transposed with an overline) $$ \bbox[lightyellow] { \overline {{\bf T}_{\,{\bf 1}} } \;{\bf T}_{\,{\bf 1}} = \overline {{\bf T}_{\,{\bf 2}} } \;{\bf T}_{\,{\bf 2}} } \tag{1} $$ which provides $6$ quadratic equations in the $6$ unknowns $\lambda_{1,k}$, $\lambda_{2,j}$, and where the sign of the solutions (if any) shall be chosen as to give that the determinant of ${\bf T}_{\,{\bf 1}} $ equals that of ${\bf T}_{\,{\bf 2}} $.

The diagonal equations provide a relation between $\lambda_{1,k}$ and $\lambda_{2,k}$ , with $k=1 \cdots 3$, of the type $$ \lambda _{\,1,\,\,k} = a_{\,k} \pm \sqrt {b_{\,k} \,\lambda _{\,2,\,\,k} ^{\,2} + c_{\,k} \,\lambda _{\,2,\,\,k} + d_{\,k} \,} $$ Replacing these values into the three off-diagonal equations, produces equations involving radicals whose solution is quite complicated.
But I cannot see at the moment a better way.

Once the $\lambda$s have been determined, an efficient way to find the rotation matrix could be that of determining
- first the rotation that brings the centroid of $T_2$ onto that of $T_1$ (around the axis normal to both)
- secondly, the rotation about the common central axis that brings one vertex of $T_2$ onto the corresponding vertex of $T_1$.

To recapitulate and try to make it more clear, the process is:

1. given the two sets of three lines each (however the lines be mutually placed within each set: skew, coincident, ..) we look for the position of a point in each line (the $\lambda$), such that the triangles resulting in each set ($T_1$ and $T_2$) be "congruent" under a rotation around the origin;
2. according to the question, as worded, congruent means that upon rotation around the origin they shall become coincident vertex $i$ over vertex $i$, excluding reflection;
3. a necessary and sufficient condition for that to occur is that, considering the pyramids $OT_1$ and $OT_2$, based on each triangle and with the vertex at the origin, the lengths of the corresponding edges from the origin be equal and the angles between corresponding pairs of those edges be equal, i.e. be equal their scalar product;
4. such condition is summarized in the matrix equation (1), where each side contains the scalar product of each edge with itself and with the others, therefore the matrices are symmetric and result in $6$ quadratic equations in the $6$ parameters;
5. note that, if only superposition of the triangles be required (allowing permutation), then in equation (1) we shall add the OR of the cases in which $T_2$ is permuted, but only with permutation matrices of det $+1$ if pure rotation without reflection is allowed;
6. once the parameters are assigned with the values (if existing) making the triangles congruent as per above, then the whole pyramids are congruent (equal edges, angles, areas and volume), and when the triangles are brought to coincide, the pyramids will also coincide, keeping firm their vertex at the origin, i.e. by a rigid rotation around the origin;
7. such a rotation can be found by separating it into a rotation that brings one edge (or the centroid axis) onto the corresponding one, followed by a rotation around such common edge/axis to get the coincidence of other two edges.

Concerning the solution approach to eq. (1) refer to the related post

Solution 2:

Writing the paramtric equations of the lines, with respective parameters $r_i$ and $s_j$, you have

$$R\,\,(\mathbb p_i+r_i\mathbb v_i)=\mathbb q_j+s_j\mathbb u_j$$ where $i,j$ are some (possibly distinct) permutations of $\{0,1,2\}$ (so that you will have to solve the problem six times).

Rewriting

$$R\,\mathbb p_i+r_iR\,\mathbb v_i=q_j+s_j\mathbb u_j$$ and taking the cross product with $(R\,\mathbb v_i)\times\mathbb u_j$, you can get rid of the parameters with

$$R\,\mathbb p_i\times((R\,\mathbb v_i)\times\mathbb u_j)=q_j\times((R\,\mathbb v_i)\times\mathbb u_j).$$

Then using the expression of $R$,

$$R=R_x\cdot R_y\cdot R_z,$$ you get a nasty system of nine nonlinear (trigonometric) equations in three unknows.

The equations can also be cast in terms of the matrix elements, which you constrain by expressing the orthonormality of the matrix,

$$R^TR=I.$$

Hence you get a monstrous system of fifteen quadratic equations in nine unknowns, which can have up to $512$ distinct solutions, if I am right, and will be subject to compatibility conditions.