Finding the volume of $z^2<4xy$ if $x,y,z \in [0,1]$.

enter image description here

As clarified, your question is to find the volume of the region inside the cube $[0, 1]^3$ below $z^2 = 4xy$. One way is to first find the volume bound above the unit square $[0, 1]^2$ in xy-plane and then subtract the volume above the unshaded part and above $z = 1$.

So the integral will be,

$ \displaystyle V_1 = \int_0^1 \int_0^1 2 \sqrt{xy} ~dx ~dy$

$\displaystyle V_2 = \int_{1/4}^1 \int_{1/(4y)}^1 (2 \sqrt{xy} - 1) ~dx ~dy$

And finally $V = V_1 - V_2$