How can we show the cone $x^2 +y^2 = z^2$ is not a smooth manifold?

Let us use your "preliminary" definition of a $k$-manifold in ${\mathbb R}^n$. You are given the set $$S:=\{(x,y,z)\in{\mathbb R}^3\ |\ f(x,y,z)=0\},\qquad f(x,y,z):=x^2+y^2-z^2\ .$$ As you have remarked, at any point ${\bf p}\in S$ with ${\bf p}\ne{\bf 0}$ one has $\nabla f({\bf p})\ne{\bf 0}$. By the implicit function theorem in the neighborhood of such ${\bf p}$ the set $S$ satisfies your manifold condition with $k=2$.

Remains the point ${\bf p}={\bf 0}\in S$, and we stick to $k=2$. Selecting $I=\{1,2\}$ we obtain for each $(x,y)$ near ${\bf 0}$, but $\ne{\bf 0}$, two values $z$ such that $f(x,y,z)=0$, namely $z=\pm\sqrt{x^2+y^2}$. It follows that this $I$ does not qualify. Trying $I=\{1,3\}$, i.e., $x$ and $z$ as independent variables, we obtain $$y=\pm\sqrt{z^2-x^2}\ ,$$ which is even undefined in half of any neighborhood of $(0,0)$. Same thing with $I=\{2,3\}$.

It follows that at ${\bf 0}$ the manifold character of $S$ is definitively defect.


Another hint: If you remove a point from a connected topological manifold of dimension $>1$, the result is still connected. Now, look at your quadric and try to figure out which point disconnects it.


Hint. Is the graph of $x\mapsto |x|$ a smooth manifold? It can be described as $\{(x,z)\ |\ x^2=z^2, z\geq 0\}$.

Update on the request of the OP:

The graph is not a smooth manifold, because:

  1. it is not (near $0$) a graph of the variable $y$, because any of the reasons:

    1) there are two values of $x$ for one $y$,

    2) the set $V$ in your definition must be required be open (it is a mistake), and it can't be open in this case, because $y$ is nonnegative.

  2. it is not a $C^\infty$ graph of $x$ (certainly it is a graph of $x\mapsto |x|$ [thus can't be a graph of any other function of $x$] and certainly this function isn't $C^\infty$).