Solution of a simultaneous equations in $a,b,c$

Solution 1:

If you expand, you get $$(a + b)(b + c)(c + a) = a^2b + b^2c + c^2a + ab^2 + bc^2 + ca^2 + 2abc.$$ This can be "simplified" into $$(ab + bc + ca)(a + b + c)-abc = 20 - abc$$ using the equations you determined. Clearly, then, we want to maximize $abc$. Notice that $a, b,$ and $c$ are the roots of the polynomial equation $$x^3 - 4x^2 + 5x - abc = 0.$$ This equation must have all real roots, so its discriminant must be non-negative: \begin{align*} (-4)^2(5^2) - 4\cdot5^3 - 4 \cdot(-4)^3(abc) - 27(abc)^2 + 18(-4)(5)(abc) &\geq 0\\ \Longrightarrow -27(abc)^2 - 104(abc) - 100 &\geq 0\\ \Longrightarrow (2 - abc)(27abc - 50) &\geq 0. \end{align*} Thus, $$\frac{50}{27} \le abc \le 2,$$ so the maximum value of $abc$ is $2$, yielding a minimum of $18$ for the desired expression. Furthermore, the minimum value of $abc$ is $\frac{50}{27}$, yielding a maximum of $$20 - \frac{50}{27} = \frac{190}{27}$$ forthe desired expression.