calculating triangle coordinates from 3 lengths and its euler rotation in 3d space

Any transformation is decomposable into translation, deformation (here only isotropic deformation is present, due to equilaterality) and rotation. Rotation is done by three subsequent rotations with respect to 3 different axes, where the transformation order matters. The triangle after transformation is given by formula: \begin{align} \left[\begin{array}{c}x_i\\y_i\\z_i\end{array}\right]=\left[\begin{array}{c}X_0\\Y_0\\Z_0\end{array}\right]+ \underbrace{\frac{\text{len}_2}{\text{len}_1} \left[\begin{array}{ccc} c_3 & s_3 & 0 \\ -s_3 & c_3 & 0 \\ 0 & 0 & 1 \\ \end{array}\right] \left[\begin{array}{ccc} c_2 & 0 & s_2 \\ 0 & 1 & 0 \\ -s_2 & 0 & c_2 \\ \end{array}\right] \left[\begin{array}{ccc} 1 & 0 & 0 \\ 0 & c_1 & s_1 \\ 0 & -s_1 & c_1 \\ \end{array}\right]}_{\mathbf{A}} \left[\begin{array}{c}a_i\\b_i\\0\end{array}\right],~~i=1,2,3, \end{align} where $\text{len}_2<\text{len}_1,~z_i>0,~c_i=\cos(\theta_i),~s_i=\sin(\theta_i),~i=1,2,3$, $~\theta_1=\theta_x,~\theta_2=\theta_y,~\theta_3=\theta_z$, where matrix $\mathbf{A}$ is known and where the vector $[X_0,~Y_0,~Z_0]^{\text{T}}$ represents translation.

Continuation:

Moreover we have known distances $L_1,L_2,L_3$ so that we want \begin{align} &(x_i-a_i)^2+(y_i-b_i)^2+z_i^2=L_i^2,~~i=1,2,3, \end{align} and \begin{align} &z_i>0,~~i=1,2,3. \end{align} Also we have 12 equations \begin{align} &\left[\begin{array}{c}x_i\\y_i\\z_i\end{array}\right]=\left[\begin{array}{c}X_0\\Y_0\\Z_0\end{array}\right]+ \mathbf{A}\left[\begin{array}{c}a_i\\b_i\\0\end{array}\right],~~i=1,2,3,~~~~~~~~~~~~~~~(1)\\ &(x_i-a_i)^2+(y_i-b_i)^2+z_i^2=L_i^2,~~i=1,2,3,~~~~~~(2)\\ \end{align} for 12 unknowns $X_0,Y_0,Z_0,$ $x_i,y_i,z_i,i=1,2,3$. So we can hope, at least in some cases, in a unique solution. If this solution satisfies $z_i>0,~i=1,2,3$ then we are done. The following is however clear:

1. There are such input parameters, that problem (1,2) has no solution
2. In case $\text{len}_2=\text{len}_1$ there are such input parameters, that problem (1,2) has infinitely many solutions

Proof: 1. for $\mathbf{A}=\mathbf{0},~L_1=L_2=L_3=0$ and different points $P_1,~P_2,~P_3$ the problem definitely has no solution. 2. for $\mathbf{A}=\mathbf{I},~L_1=L_2=L_3=1$ we see that for any $X_0,Y_0,Z_0$ s.t. $X_0^2+Y_0^2+Z_0^2=1$ we get solution, which satifies Eq.(2), ergo we have infinitely many different nontrivial solutions in this case. $\clubsuit$

Remark (uniqueness in case len2<len1): If we insert Eqs.(1) into Eqs.(2) we can solve instead of 12 equations only 3 nonlinear equations: \begin{align} &(X_0+(a_{11}-1)a_i+a_{12}b_i)^2+(Y_0+a_{21}a_i+(a_{22}-1)b_i)^2+\\&(Z_0+a_{31}a_i+a_{32}b_i)^2=L_i^2,~~i=1,2,3.~~~~~~(3)\\ \end{align} This is a problem of intersection of three spheres in $\mathbb{R}^3$, which has in general $0,1,2$ solutions (infinite number of solutions are not possible in case len2<len1). If we can proof, that in case of 2 different solutions one yields negative values of $z_i$ (which is indicated by all numerical solutions, which I have tried), then the proof of uniqueness is made. $\diamondsuit$

Rotating the vector $\vec u$ for $\theta$ regarding a direction $\vec v$ with $\|\vec u\|=1,\|\vec v\|=1$ and $\vec u\cdot\vec v = 0$ we can build an equilateral triangle using the Rodrigues formula

$$ \vec u_{\theta} = R(\vec v,\theta)\cdot \vec u = \vec u\cos(\theta)+\vec v\times\vec u\sin(\theta)+\vec v\cdot\vec u(1-\cos(\theta))\vec v $$

so giving a vertex $r_1\vec u$ we can construct the other two as follows

$$ T(\theta_1,r_1,\vec u,\vec v,\vec o) = \vec o+r_1\left(R\left(\vec v,\theta_1\right)\cdot\vec u,R\left(\vec v,\theta_1+\frac 23\pi\right)\cdot \vec u,R\left(\vec v,\theta_1+\frac 43\pi\right)\cdot \vec u\right) $$

Here $\vec o$ represents the triangle's barycenter. This triangle is submitted to a Euler matrix $E(\alpha_0,\beta_0,\gamma_0)$ rotation as follows

$$ E(\alpha_0,\beta_0,\gamma_0)\cdot T(\theta_1,r_1,\vec u,\vec v,\vec o) = \mathbb{T}(\theta_1,r_1,\vec u,\vec v,\vec o) $$

Now giving a known triangle $T_0(\psi_0,r_2,\vec u_0,\hat{e}_z,\vec o_0)$ with $\psi_0,r_2$ given and $\hat{e}_z=(0,0,1),\vec u_0 = (1,0,0),\vec o_0=(1,1,0)$ we can define as objective function

$$ \mathcal{O}\left(\theta_1,\vec u,\vec v,\vec o\right)=\sum_k \|\|\mathbb{T}(\theta_1,r_1,\vec u,\vec v,\vec o)-T_0(\psi_0,r_2,\vec u_0,\hat{e}_z,\vec o_0)\|_k-\delta_k\| $$

Here $\delta_k$ is given and the $\|\cdot\|_k$ is taken at each corresponding vertex in the triangles.

Finally, the determination of $T(\theta_1,r_1,\vec u,\vec v,\vec o)$ can be handled by solving the minimization problem

$$ \min_{\theta_1,\vec u,\vec v,\vec o}\mathcal{O}\left(\theta_1,\vec u,\vec v,\vec o\right)\ \ \ \text{s. t.}\ \ \ \cases{\|\vec u\|=1\\ \|\vec v\|=1\\ \vec u\cdot\vec v = 0\\ \mathbb{T}(\theta_1,r_1,\vec u,\vec v,\vec o)_k\cdot\hat{e}_z\ge 0} $$

Depending on the values for $\alpha_0,\beta_0,\gamma_0$ the problem my not have solution or have non unique solutions.

Follows a case study with two solutions depicted in the attached graphic

NOTE

Here in green the triangle $T_0$ with $\psi_0 = 0,r_2 = 13,\hat{e}_z=(0,0,1),\vec u_0 = (1,0,0),\vec o_0=(1,1,0)$ and $\delta = (41.79, 27.6849, 44.9287)$ and for the rotation $\alpha_0 = 0.349093,\beta_0=-0.613477,\gamma_0=-0.0460201$ we have compatibility with the two solutions $r_1 = 10, \cases{\theta_1 =\frac{\pi}{4}, \vec u =(1,0,-1)/\sqrt{2}, \vec v = (1,1,1)/\sqrt{3},\vec o = (20,30,10)\\ \theta_1 = -3.0919,\vec u = (0.0914184,-0.550551,0.829781), \vec v = (0.76595,0.571371,0.294713),\vec o = (18.24,32.1887,2.45912) }$

In red are the two versions of $T_1$ and in blue the rotated versions.