Factorize $abx^2-(a^2+b^2)x+ab$

Factorize the quadratic trinomial $$abx^2-(a^2+b^2)x+ab.$$ The discriminant of the trinomial is $$D=(a^2+b^2)^2-(2ab)^2=\\=(a^2+b^2-2ab)(a^2+b^2+2ab)=(a-b)^2(a+b)^2=\\=\left[(a-b)(a+b)\right]^2=(a^2-b^2)^2\ge0 \text{ } \forall a,b.$$ So the roots are $$x_{1,2}=\dfrac{a^2+b^2\pm\sqrt{(a^2-b^2)^2}}{2ab}=\dfrac{a^2+b^2\pm\left|a^2-b^2\right|}{2ab}.$$ How can I expand the modulus here? Have I calculated the discriminant in a reasonable way? Can we talk about "the discriminant of a quadratic trinomial" or only the corresponding quadratic equation (the trinomial=0) has a discriminant? What about "the roots of a trinomial" (or of the corresponding quadratic equation)?

Using the quadratic formula is a good approach, and everything you have done so far is correct. The modulus/absolute value can be replaced by normal brackets as there is a $\pm$ preceding it, and the numbers involved here are real. Notice that:

$$\frac {a^2 +b^2 + (a^2 - b^2)}{2ab} = \frac ab, \quad \frac {a^2 +b^2 - (a^2 - b^2)}{2ab} = \frac ba$$

hence the roots are $\dfrac ab$ and $\dfrac ba$, and we are looking at the factors $(bx- a)$ and $(ax - b)$. Now:

$$(bx-a)(ax- b) = abx^2 - a^2 x - b^2 x + ab$$

which is exactly what we are trying to factorize.

For the title question. The factorization is given by $$ (ax-b)(bx-a)=abx^2-(a^2+b^2)x+ab. $$ We can find it by multiplying out $(\rho_1x+\rho_2)(\rho_3x+\rho_4)$ and comparing coefficients. This is easier than computing a discriminant.

Using the Quadratic Formula is more general but here is an alternative method:

We have $abx^2-(a^2+b^2)x+ab=0 \iff x^2-\frac{a^2+b^2}{ab}x+1=0 \iff x^2-(\frac{a}{b}+\frac{b}{a})x+1=0$.

Let $r_1$ and $r_2$ be the (possibly equal, possibly complex) roots of the quadratic function $x^2-(\frac{a}{b}+\frac{b}{a})x+1$. Then their product is $1$ and their sum is $\frac{a}{b}+\frac{b}{a}$. This shows that the roots are $\frac{b}{a}$ and $\frac{a}{b}$.

Therefore, $x^2-(\frac{a}{b}+\frac{b}{a})x+1=(x-\frac{a}{b})(x-\frac{b}{a})$ so

$$abx^2-(a^2+b^2)x+ab = ab(x-\frac{a}{b})(x-\frac{b}{a}) = (ax-b)(bx-a)$$

Everything is OK.

First: Accept

$$a^2≥b^2$$

Then accept

$$a^2≤b^2$$

You will see that the roots have not changed.

We have

$$abx^2-(a^2+b^2)x+ab=(a x - b) (b x - a)$$

Small Supplement:

Note that,