Transform or transformation?

Solution 1:

In the particular domain you're referencing, both transform and transformation have an established history of usage.

For example, we speak of the Hough transform or the Fourier transform. We also talk about affine transformations or homothetic transformations.

I'm not aware of any specific rules, but in general, I've noticed that "named" objects tend to be referenced as the foo transform, while unnamed or otherwise generic ones use the foo transformation.

In this case, I would probably use transformation since it appears you're referring to a generic transformation matrix. So:

To recognize with the optimized model, first the transformation is applied and then the likelihoods are calculated.

Solution 2:

As @Dusty said, this is relevant to your specific domain...

However, I felt that there was a more specific difference:

You use a transform to perform a transformation.

That is, if you're outlining the model of which you speak it would be "transform".
However, if you're referring to the actual effect (in runtime, so to speak), its transformation.

Solution 3:

In my opinion, it is more convenient to use "transformation". One of the reasons is "transform" is used usually as a verb, and rarely as a noun. For example, we say "linear transformation" instead of saying "linear transform". Another reason is "transform (noun)" was first used in 1853 [¹] whereas "transformation" was first used in 15th century [²]. So I would prefer:

To recognize with the optimized model, first the transformation is applied and then the likelihoods are calculated.

Solution 4:

I'm very familiar with using the word transform in the mathematical domain. In you case, you should use the word transformation because you are "applying" it. Transform is a result of a transformation.