The unique closed orbit in GIT quotient fibers for polynomial actions of Gl
The following reasoning must contain a flaw somewhere because I end up with something absurd, and I cannot figure out where the mistake is. I hope that someone can point it out to me.
Let $M$ be the algebraic monoid of $n\times n$-matrices and $G$ its unit group, i.e. the subgroup of invertible matrices. Let $X$ be a $G$-variety. Since $G$ is reductive, we can consider the quotient $\pi:X\to\newcommand{\qq}{\mathop{/\hspace{-2.5pt}/}}X\qq G$. It is well-known (see, e.g., Proposition 27.5.3 in the book Lie algebras and algebraic groups by Tauvel and Yu) that for $x\in X$, the fiber $\pi^{-1}(\pi(x))$ contains a unique closed orbit $O$ and $$\pi^{-1}(\pi(x)) = \{ y\in X \mid O\subseteq\overline{G.y} \}.$$
Now let us assume that the action of $G$ extends to an action of $M$ on $X$. For instance, this is the case when $X$ is a $G$-module on which $G$ acts polynomially. It is then easy to see that for any $x\in X,$ we have $G.x\subseteq M.x\subseteq \overline{G.x}$:
Let $\alpha_x:M\to X$ be the action morphism $a\mapsto a.x$. Let $U:=\alpha_x^{-1}(G.x)$. We certainly have $G\subseteq U \subseteq M$. As $G$ is open in $M$ and $M$ is irreducible, $\overline{U}=M$. Furthermore, $\alpha_x^{-1}(\overline{G.x})$ is a closed set containing $U$. Thus, we must have $M=\alpha_x^{-1}(\overline{G.x})$.
Now for any $x\in X$, we can consider $y:=0.x$, i.e. the image of $x$ under the action of the zero matrix. By the above, we have $y\in\overline{G.x}$. On the other hand, we have $G.y=\{y\}$, so $y$ is a closed $G$-orbit.
Now I can think of many examples where $X$ is a linear space and $0.x=0$ for any $x\in X$. In this case, $0$ would be a closed $G$-orbit contained in the orbit closure of any point, so $\pi^{-1}(\pi(0))=X$. In other words, $X\qq G=\{\ast\}$.
That is completely absurd. Where is my mistake?
Edit. To clarify what I mean by the quotient $X\qq G$, let us assume $X$ to be affine throughout. Then, I define $X\qq G:=\mathrm{Spec}(\Bbbk[X]^G)$ as the spectrum of the ring of $G$-invariants of the coordinate ring of $X$. There is a fundamental result (by Hilbert, I believe) that states that $\Bbbk[X]^G$ is actually a finitely generated $\Bbbk$-algebra, so this is well-defined.
There is no error. If $X$ is a $G$ representation which extends to an $M$ action, with $0 \cdot x = 0$ for any $x \in X$, then $k[X^G]=k$. For example, let $X$ be $\mathbb{A}^d$ and $G$ is $GL_1$, acting by $t \cdot (x_1, x_2, \ldots, x_d) = (t x_1, t x_2, \ldots, t x_d)$. Then $k[x_1, \ldots, x_d]^G = k$ and $X/G = \{ \ast \}$. Your proof is correct.
I think the problem is that you think that the points of $X/G$ to be something like the $G$-orbits. For example, perhaps you are expecting to get $\mathbb{P}^{d-1}$ in the above example. But this is not always true, and this is a good example of it. In order to get the points of $X/G$ to be something like the $G$-orbits, you have to remove the "unstable" points. In the above example, you have to remove the origin in order to get the quotient to be $\mathbb{P}^{d-1}$. I'd suggest reading up on Geometric Invariant Theory for more.