Solve $X$ from $AXA^{-1}T = BXB^{-1}$

First of all, we can simplify the equation somewhat. If we make the substitution $Y = AXA^{-1}$, then this equation can be written as $$ YT = (BA^{-1})Y(BA^{-1})^{-1} \implies YT = CYC^{-1}, $$ where $C = BA^{-1}$.

From there, I do not believe that there is a nice "closed form solution" for $X$. However, a solution can be obtained using vectorization. Note that each element of $SE(3)$ can be represented with an augmented matrix. In particular, the map that associated the affine transformation $T(x) = Ax + b$ with the augmented matrix $$ [T] = \pmatrix{A & b\\0 & 1} $$ is an injective Lie group homomorphism. Thus, we may replace each of the elements $C,T,X$ with their associated $4 \times 4$ matrices, so that $YT = CYC^{-1}$ is a linear equation over the entries of the matrix $Y$.

With vectorization, we can see that the equation can be expressed as $$ (T^\top \otimes I_4) \operatorname{vec}(Y) = (C^{-\top} \otimes C)\operatorname{vec}(X) \implies\\ [T^\top \otimes I_4 - C^{-\top} \otimes C]\operatorname{vec}(X) = 0, $$ where $I_4$ denotes the size $4$ identity matrix.

We can limit the solution set among the set of all $4 \times 4$ matrices to only those corresponding to elements of $SE(3)$ by applying the following additional constraints:

  • The last row of $Y$ must be $(0,0,0,1)$,
  • The upper left $3 \times 3$ matrix $M$ of $Y$ must be an element of $SO(3)$, which is to say that $M^TM = I_3$ with $\det(M) = 1$.

Rewrite the equation as $X(A^{-1}TB)=(A^{-1}B)X$ or, in block form, $$ \pmatrix{R&v\\ 0&1}\pmatrix{R_1&v_1\\ 0&1}=\pmatrix{R_2&v_2\\ 0&1}\pmatrix{R&v\\ 0&1}. $$ Multiply out and simplify, we get two equations $RR_1=R_2R$ and $(I-R_2)v=v_2-Rv_1$. Once $R$ is determined from the first equation, the second equation is just a non-homogeneous linear equation in $v$. Since $R_2$ is a $3\times3$ rotation matrix, $I-R_2$ is necessarily singular. Thus the second equation has either no solution or infinitely many solutions. In particular, if $v$ is a solution, then $v+u$ is also a solution for every vector $u$ lying on the rotation axis of $R_2$.

The similar can also be said to the first equation. Let $q=w+xi+yj+zk,\ q_1=w_1+x_1i+y_1j+z_1k$ and $q_2=w_2+x_2i+y_2j+z_2k$ be the unit quaternion representations of $R,R_1$ and $R_2$ respectively. The equation $RR_1=R_2R$ can then be rewritten as $$ \pmatrix{ w_1-w_2&x_2-x_1&y_2-y_1&z_2-z_1\\ x_1-x_2&w_1-w_2&z_1+z_2&-(y_1+y_2)\\ y_1-y_2&-(z_1+z_2)&w_1-w_2&x_1+x_2\\ z_1-z_2&y_1+y_2&-(x_1+x_2)&w_1-w_2 }\pmatrix{w\\ x\\ y\\ z}=0 $$ and we are looking for a unit vector solution $(w,x,y,z)$. Since the $4\times4$ matrix on the left is in the form of $(w_1-w_2)I+K$ for some skew-symmetric matrix $K$, the multiplicity of the zero eigenvalue, if any, is even. It follows that the equation has either no unit vector solution (this occurs, e.g., when $R_1=I_3\ne R_2$) or infinitely many solutions.