Problem superimposing and aligning 3D triangles

I am trying to superimpose two 3D triangles for a molecular modeling problem. It seemed simple enough. I translated the first point of each triangle to the origin, 0,0,0. I then calculated the angle I would have to rotate around the z axis to put the second point on the x axis. Using the formula for x,y,z for Rz(theta) this would be the angle where y=0, y=xsin(theta)+ycos(theta)=0, and rearranging, tan(theta)=-y/x The angle would be the arctan(-y/x). But, plugging this value for the angle back into the original equation above does not give zero except in the case where x=y, and the tangent is one. Seems like simple algebra - why doesn't this work? Thanks for any help.

As the other comments suggested you most likely got confused withing projections and goniometrics. There is also safer way without goniometrics using vector math (linear algebra).

1. create transform matrix m0 representing aligned plane to first triangle t0

   by aligned I mean one of the edges of triangle should lie in the one of the plane basis vectors. That is simple you just set one basis vector as the edge in question, origin as one of its point and exploit the cross product to get the remainding vectors.

   so if our triangle has points p0,p1,p2 and our basis vectors are x,y,z with origin o then:

   x = p1-p0;      x /= |x|;
   y = p2-p0; 
   z = cross(x,y); z /= |z|;
   y = cross(z,x); y /= |y|;
   o = p0
   

   so just feed those to the transform matrix (see the link in bottom of answer)

2. create transform matrix m1 representing aligned plane to second triangle t1

   its the same as #1

3. compute final transform matrix m converting t1 to t0

   that is simple:

   m = Inverse(m1)*m0
   

Now any point from t1 can be aligned to t0 simply by multiplying m matrix by the point. Do not forget to use homogenuous coordinates so point(x,y,z,1)

Here small C++/OpenGL example:

//---------------------------------------------------------------------------
double t0[3][3]=    // 1st triangle
    {
    -0.5,-0.5,-1.2,
    +0.5,-0.5,-0.8,
     0.0,+0.5,-1.0,
    };
double t1[3][3]=    // 2nd triangle
    {
    +0.5,-0.6,-2.1,
    +1.5,-0.5,-2.3,
    +1.2,+0.3,-2.2,
    };
double arot=0.0; // animation angle
//---------------------------------------------------------------------------
void gl_draw()      // main rendering code
    {
    int i;
    double m0[16],m1[16],m[16],x[3],y[3],z[3],t2[3][3];

    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
    glDisable(GL_CULL_FACE);
    glEnable(GL_DEPTH_TEST);

    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();
    glTranslated(0.0,0.0,-10.0);
    glRotatef(arot,0.0,1.0,0.0); 

    // render original triangles
    glBegin(GL_TRIANGLES);
    glColor3f(1.0,0.0,0.0); for (i=0;i<3;i++) glVertex3dv(t0[i]);
    glColor3f(0.0,0.0,1.0); for (i=0;i<3;i++) glVertex3dv(t1[i]);
    glEnd();

    // x,y,z = t0 plane basis vectors
    vector_sub(x,t0[1],t0[0]);  // x is fisrt edge
    vector_one(x,x);            // normalized
    vector_sub(y,t0[2],t0[0]);  // y is last edge
    vector_mul(z,x,y);          // z = cross(x,y) ... perpendicular vector to x,y
    vector_one(z,z);
    vector_mul(y,z,x);          // y = cross(z,x) ... perpendicular vector to z,x
    vector_one(y,y);
    // m0 = transform matrix representing t0 plane
    m0[ 3]=0.0; for (i=0;i<3;i++) m0[ 0+i]=x[i];
    m0[ 7]=0.0; for (i=0;i<3;i++) m0[ 4+i]=y[i];
    m0[11]=0.0; for (i=0;i<3;i++) m0[ 8+i]=z[i];
    m0[15]=1.0; for (i=0;i<3;i++) m0[12+i]=t0[0][i];

    // x,y,z = t1 plane basis vectors
    vector_sub(x,t1[1],t1[0]);  // x is fisrt edge
    vector_one(x,x);            // normalized
    vector_sub(y,t1[2],t1[0]);  // y is last edge
    vector_mul(z,x,y);          // z = cross(x,y) ... perpendicular vector to x,y
    vector_one(z,z);
    vector_mul(y,z,x);          // y = cross(z,x) ... perpendicular vector to z,x
    vector_one(y,y);
    // m1 = transform matrix representing t1 plane
    m1[ 3]=0.0; for (i=0;i<3;i++) m1[ 0+i]=x[i];
    m1[ 7]=0.0; for (i=0;i<3;i++) m1[ 4+i]=y[i];
    m1[11]=0.0; for (i=0;i<3;i++) m1[ 8+i]=z[i];
    m1[15]=1.0; for (i=0;i<3;i++) m1[12+i]=t1[0][i];

    // m = transform t1 -> t0 = Inverse(m1)*m0
    matrix_inv(m,m1);
    matrix_mul(m,m,m0);

    // t2 = transformed t1
    for (i=0;i<3;i++) matrix_mul_vector(t2[i],m,t1[i]);

    // render transformed triangle
    glLineWidth(2.0);
    glBegin(GL_LINE_LOOP);
    glColor3f(0.0,1.0,0.0); for (i=0;i<3;i++) glVertex3dv(t2[i]);
    glLineWidth(1.0);
    glEnd();

    glFlush();
    SwapBuffers(hdc);
    }
//---------------------------------------------------------------------------

I used my own matrix and vector math hope the comments are enough if not see:

• Understanding 4x4 homogenous transform matrices

For info about the matrices and you will find the sources and equations for the math used there too. Here preview for my test case:

Where red is t0 triangle blue is t1 triangle and green is the m*t1 transformed triangle. As you can see no need for goniometrics/euler angles at all. I rotate the stuff by arot just to visually check if the green triangle really align to the blue to prove me I did not make a silly mistake.

Now its unclear how exactly you want to align, so for example if you want maximal coverage or something either try all 3 combinations and remember the best or align to closest or largest edges of both triangles etc ...