Min $P=\frac{x^5y}{x^2+1} +\frac{y^5z}{y^2+1} +\frac{z^5x}{z^2+1}$

There's no minimum of $P$ at all. Consider an intersection of $x^2y+y^2z+z^2x = 3$ and $y = x^{-4}$, i.e. $x^{-2} + x^{-8}z + z^2x = 3$ or $x^9 z^2 + z + x^6 - 3x^8 = 0$. We can find from here that there exists a positive solution $$ z = \dfrac{-1 + \sqrt{1+12x^{17}-4x^{15}}}{2x^9}. $$ at least for $x$ big enough. Moreover, $$ \lim_{x \to +\infty}z\sqrt{x} = \sqrt{3}. $$ Hence $$ \lim_{x \to +\infty}\left[ \dfrac{x^5 y}{x^2+1} + \dfrac{y^5 z}{y^2+1} + \dfrac{z^5 x}{z^2+1}\right] = \lim_{x \to +\infty}\left[\dfrac{y^5 z}{y^2+1} + \dfrac{z^5 x}{z^2+1}\right] = \lim_{x \to +\infty}\dfrac{z^5 x}{z^2+1} = \lim_{x \to +\infty}\dfrac{3^{\frac{5}{2}}x^{-\frac{5}{2}} x}{3 x^{-1}+1} = 0. $$ So $P$ does not have a minimal value on the set $(x,y,z) \in (0, +\infty) \times (0, +\infty) \times (0, +\infty)$ (because $P$ is obviously positive on it).

Too long for a comment.

Considering the minimization problem

$$ \max_{x,y,z} -f(x,y,z)\ \ \text{s. t.}\ \ g(x,y,z) = 0 $$

with

$$ \cases{ f(x,y,z) = \frac{x^5 y}{x^2+1}+\frac{x z^5}{z^2+1}+\frac{y^5 z}{y^2+1}\\ g(x,y,z) = x^2 y + y^2 z + z^2 x - 3 } $$

and according to the notes, the point $x=y=z=1,\lambda=\frac 56$ is a stationary point on the lagrangian

$$ L=-f+\lambda g $$

This point can be qualified using the bordered hessian which is

$$ H = \left( \begin{array}{cccc} 0 & 2 x y+z^2 & x^2+2 y z & 2 x z+y^2 \\ 2 x y+z^2 & -\frac{2 x^3 \left(3 x^4+9 x^2+10\right) y}{\left(x^2+1\right)^3} & -\frac{x^4 \left(3 x^2+5\right)}{\left(x^2+1\right)^2} & -\frac{z^4 \left(3 z^2+5\right)}{\left(z^2+1\right)^2} \\ x^2+2 y z & -\frac{x^4 \left(3 x^2+5\right)}{\left(x^2+1\right)^2} & -\frac{2 y^3 \left(3 y^4+9 y^2+10\right) z}{\left(y^2+1\right)^3} & -\frac{y^4 \left(3 y^2+5\right)}{\left(y^2+1\right)^2} \\ 2 x z+y^2 & -\frac{z^4 \left(3 z^2+5\right)}{\left(z^2+1\right)^2} & -\frac{y^4 \left(3 y^2+5\right)}{\left(y^2+1\right)^2} & -\frac{2 x z^3 \left(3 z^4+9 z^2+10\right)}{\left(z^2+1\right)^3} \\ \end{array} \right) $$

and at the point of interest we have

$$ H(1,1,1) = \left( \begin{array}{cccc} 0 & 3 & 3 & 3 \\ 3 & -\frac{11}{2} & -2 & -2 \\ 3 & -2 & -\frac{11}{2} & -2 \\ 3 & -2 & -2 & -\frac{11}{2} \\ \end{array} \right) $$

now according to the notes, $n = 3, m = 1$ so we need to consider the two minors $H_3$ and $H_4$. Following we have $\det(H_3) = 63 > 0$ and $\det(H_4) = -\frac{1323}{4} < 0$. The signs are $\text{sign}(\det(H_3)) = (-1)^{m+1} = 1$ and $\text{sign}(\det(H_4))$ should alternate as negative.

This characterizes the point $(x,y,z)=(1,1,1)$ as a local minimum for $f$ constrained.

NOTE

Taking from the restriction $x$ as

$$ x = \phi(y,z) = \frac{\sqrt{z^4-4 y^3 z+12 y}-z^2}{2 y} $$

and substituting into $f(x,y,z)$ we obtain a surface

$$ u = f(\phi(y,z),y,z) $$

which can be depicted from the following plot. In blue the relative minimum point with coordinates $u=\frac 32, y=1,z=1$