Convexity, Hessian matrix, and positive semidefinite matrix

I guess the problem is with how you have approached $\vec{x}^{T}H\vec{x} \ge 0$. In this equation, you wish to find whether matrix H is positive definite or not…..

While doing so, the elements of $\vec{x}$ has to be independent of elements of matrix $H$. I mean, when you use this equation to test the definiteness of the matrix, the elements of $H$ are constants and elements of vector $\vec{x}$ are variables. What you have dones is you have took the vector $\vec{x}$ as variable $x$ at position $(1,1)$ and variable $y$ at position $(2,1)$ [mind that the elements of matrix H are also in terms of variables x and y]. If your vector $\vec{x}$ were of other variables, say $\vec{x} = (p,q)$, your answer will be fine.

First, you had $$x^{T}Hx=6x^{2}y^{-4}$$.

$$x$$ is a non-zero real-valued column vector. Let $$x=\begin{vmatrix} x_{1}\\ x_{2} \end{vmatrix}$$, then $$x^{T}Hx=\begin{vmatrix} x_{1} & x_{2} \end{vmatrix}\begin{vmatrix} 2y^{-4} &-8xy^{-5} \\ -8xy^{-5}& 20x^{2}y^{-6} \end{vmatrix}\begin{vmatrix} x_{1}\\ x_{2} \end{vmatrix}=2x_{1}^{2}y^{-4}+20x_{2}^{2}x^{2}y^{-6}-16x_{1}x_{2}xy^{-5}$$ It is hard to tell whether this function is positive or negative, while the only thing that is known is $$x\geq 0,y> 0$$.

Second, you got the determinant of the Hessian matrix to be $$40x^{2}y^{-10}-64x^{2}y^{-10}=-24x^{2}y^{-10}\leq 0$$ and you concluded that the function was "concave".

While the expression you had for the determinant of the Hessian is correct, your conclusion needs re-considerations.

The determinant of the first principal minor is $$2y^{-4}> 0$$, since $$y>0$$.

Then, if the determinant of the Hessian matrix is greater than $$0$$, then the function is strictly convex. If the determinant of the Hessian is equal to $$0$$, then the Hessian is positive semi-definite and the function is convex.

For the function in question here, the determinant of the Hessian is $$-24x^{2}y^{-10}\leq $$. There is a lot of uncertainty here. If $$x$$ takes on the value of $$0$$ and you have a $$0$$ determinant for the Hessian, then you have $$\Delta _{1}= 2y^{-4}> 0,\Delta _{2}=-24x^{2}y^{-10}= 0$$, this is the criteria for positive semi-definite and the function is convex.

However, if $$x$$ takes on non-zero value, then determinant of the Hessian is indeed negative. In that case, you have $$\Delta _{1}>0,\Delta _{2}< 0$$. The Hessian matrix is actually indefinite and no conclusion about the concavity (or convexity) of the function can be made from the Hessian matrix.

There are a lot of ambiguities here.