Solution 1:

Your work considers a cone with vertex at the origin.

Now if you are integrating wrt $dz$ first then for all values of $r$, the upper bound of $z$ is $h$ and the lower bound is a function of $r$. Please see the diagram at the end of the answer that shows your mistake. The integral in the numerator should be,

$ \displaystyle 2\pi\int_0^a \int_{(r h) /a}^{h} z ~r ~dz~ dr = 2\pi \cdot \frac{a^2h^2}{8}$

and in the denominator it should be,

$\displaystyle 2\pi\int_0^a \int_{(r h) /a}^{h} ~r ~dz~ dr = 2 \pi \cdot \frac{a^2h}{6}$


Below is a diagram that shows $2$D projection of the cone from side. Based on your bounds, for every value of $r \in (0, a)$, you are integrating over the vertical strip marked in red. That gives you center of mass of the cylinder of radius $a$ and height $h$ with the given cone cut out of it. That is the unshaded region in the diagram. You should instead be integrating over the blue strip.

enter image description here