A picture is worth a thousand words .. you really need a sketch of this for us to see (I'm too lazy to do this for you). But I suspect you have done exactly as you requested, rotated the triangle to match the normal. You probably need another rotation to match the orientation of the triangle. And a scaling, if you want the scales to match somehow?

Am not clear in what sense you want the triangles to match, you can match normals, match orientation of some edge, match scales according to an edge etc etc. What is the purpose? Is it for finding transformations to bolt some geometry onto an anchor?

After your algorithm only center and normals match, but you still need to rotate around the normal. E.g. using the 2 vectors VertexA0-CenterA and VertexB0-CenterB, and build axis angle rotation from them.

This assumes your triangles both are equal and just placed differently. If they are not, we could talk about how to build a proper average angle from 3 vertices, but this depends on what you try to do...

You could do the whole thing faster if you build orientation for both triangles and calculate the rotation between them, e.g.:

Matrix3x3 ornA;

ornA[0] = normalA;

ornA[1] = (VertexA0-CenterA).Normalized();

ornA[2] = ornA[0].Cross(ornA[1]); // or vice versa - depends on right / left handed convention

Matrix3x3 ornB; 

// ...same for B

Matrix3x3 rotationFromAtoB = ornA.Inveresd() * ornB; // not sure if i should inverse B instead  also multiplication order depends on convention

6 hours ago, grumpyOldDude said:

I understand most of it, except for the last step ie: ornA.Inveresd()*ornB. Can you explicitly throw a bit more light on the maths details of ornA.Inveresd()*ornB.

haha, ok... what i do in practice is: There are 4 possibilities if we combine 2 options of multiplication order and 2 options if which one to inverse, and often i end up trying all 4 of them until it works. I do so because: 1st i'm terrible at remembering answers to binary questions, and 2nd i use multiple math libs with multiple conventions, so trial and error turned up to be most effective for me personally.

Being self thought i'm not the one who can explain this very well, but i'll try:

What we want is, we want to deal with orientations and rotations in a similar easy way than we do with positions and displacements.

If we want to know the translation from one point to another, we simply do: trans = p1 - p0; so p1 = p0 + trans; This is so simple it is not worth to think about, but with rotations it seems not so simple anymore. The good news it still is, but instead subtraction to get the difference between p0 and p1, you have to use more complicated math.

Example, we have a 3x3 matrix representing some orientation, and we want to rotate some vector, say (1,0,0) so we see how it looks like using this orientation.

To do so, we would write:

Matrix3x3 orn = ... whatever

vec localV (1,0,0);

vec worldV = orn * localV;

Some other math lib would need different multiplication order, writing worldV = localV * orn;

To overcome the confusion, it would be better if we create a function with a proper name like this:

vec worldV = orn.Rotate(localV);

At this point, we could try to demystify the math we tend to abstract behind math libs. We could write:

vec worldV = orn.Xaxis() * localV.x + orn.Yaxis() * localV.y + orn.Zaxis() * localV.z;

(understanding how this works is the first necessary point to get!)

If you use matrices that store vectors in row order, we could write this using vector / element indexing like this:

vec worldV = orn[0] * localV[0] + orn[1] * localV[1] + orn[2] * localV[2];

... which makes a more obvious connection to how simple linear algebra relates to geometry.

Now, assume we want to do the opposite: We know worldV and we want to calculate how this vecor looks like in the local space of orn.

We can do this by making dot procucts like this:

localV[0] = orn[0].Dot(worldV);

localV[1] = orn[1].Dot(worldV);

localV[2] = orn[2].Dot(worldV);

(understanding how this works is the second necessary point to get - be sure to be able to imagine how this works in a geometric way.)

So, if this makes sense, after thinking of it, you could look at the resulting math and you would see it is the same as doing:

localV = orn.Inversed() * worldV;

or, because an oriantation as no scale, and all its directions are orthogonal to each other:

localV = orn.Transposed() * worldV;

which might be an important optimization! You can in fact add an Unrotate() function to your matrix class without a need to build transpose ore inversed - it's just a matter of proper indexing matrix elements:

localV = orn.Unrotate(worldV);

At this point we have the ability to relate directions between various orientations, we can rotate them from one to another and back the other way around. 

Now, remember that an orientation matrix itself is also just a collection of 3 directions, so we also have the ability to build rotations between two orientations - rotate forth and back between them however we want. It all sounds obvious, but it already has all the answers.

Nice! But in practice i'm not sure enough of this to avoid trial and error some times. I understand, but it's not solid enough - i still need to keep guessing. However it's ok for me because i know what must work if i try all the options. If you want to know better, you need a better teacher than me, and you should do yourself a favor trying to avoid mixing conventions

Looking up some code, i see it should be rotFronAtoB = ornB.Inveresd()*ornA for me, and because by rotation order seems to be really unconventional, it's likely rotFronAtoB = ornA*ornB.Inveresd() for you. This issue is the same no matter if working with quaternions or matrices.

EDIT: To make it clear, the line rotFronAtoB = ornB.Inveresd()*ornA first unrotates from space B to global space using inverse, and second rotates from global space to space A. (rotFromBtoA might be a better variable name  )

...spend some time trying the options and see if you get it to work, otherwise post your code eventually.