Proving the stabilizer is a subgroup of the group to prove the Orbit-Stabiliser theorem

You’re on the right track. By definition $G_x=\{g\in G:g\cdot x=x\}$. Suppose that $g,h\in G_x$; then $$(gh)\cdot x=g\cdot(h\cdot x)=g\cdot x=x\;,$$ so $gh\in G_x$, and $G_x$ is closed under the group operation. Moreover, $$g^{-1}\cdot x=g^{-1}\cdot(g\cdot x)=(g^{-1}g)\cdot x=1_G\cdot x=x\;,$$ so $g^{-1}\in G_x$, and $G_x$ is closed under taking inverses. Thus, $G_x$ is indeed a subgroup of $G$. To finish the proof, you need only verify that there is a bijection between left cosets of $G_x$ in $G$ and the orbit of $x$.

Added: The idea is to show that just as all elements of $G_x$ act identically on $x$ (by not moving it at all), so all elements of a left coset of $G_x$ act identically on $x$. If we can also show that each coset acts differently on $x$, we’ll have established a bijection between left cosets of $G_x$ and members of the orbit of $x$.

Let $h\in G$ be arbitrary, and suppose that $g\in hG_x$. Then $g=hk$ for some $k\in G_x$, and $$g\cdot x=(hk)\cdot x=h\cdot(k\cdot x)=h\cdot x\;.$$ In other words, every $g\in hG_x$ acts on $x$ the same way $h$ does. Let $\mathscr{G}_x=\{hG_x:h\in G\}$, the set of left cosets of $G_x$, and let

$$\varphi:\mathscr{G}_x\to\operatorname{Orb}(x):hG_x\mapsto h\cdot x\;.$$

The function $\varphi$ is well-defined: if $gG_x=hG_x$, then $g\in hG_x$, and we just showed that in that case $g\cdot x=h\cdot x$.

It’s clear that $\varphi$ is a surjection: if $y\in\operatorname{Orb}x$, then $y=h\cdot x=\varphi(hG_x)$ for some $h\in G$. To complete the argument you need only show that $\varphi$ is injective: if $h_1G_x\ne h_2G_x$, then $\varphi(h_1G_x)\ne\varphi(h_2G_x)$. This is perhaps most easily done by proving the contrapositive: suppose that $\varphi(h_1G_x)=\varphi(h_2G_x)$, and show that $h_1G_x=h_2G_x$.

We can have the claim for $(G\mid\Omega)$ which is transitive and so we get $$|G|=|\Omega||G_{\omega}|$$ To see this, put $\Omega^*=\{G_{\omega}x\mid x\in G\}$and define the following map: $$f:\Omega^*\to\Omega,\;\;f(G_{\omega}x)=\omega^x$$ You can easily verify that $f$ is well-defined and is a bijective. Both $\Omega^*$ and $\Omega$ are finite and then $$|\Omega|=|\Omega^*|=[G:G_{\omega}]$$

No put the condition of being transitive aside, you will have your own claim as others confirmed.

You should show that the identity of $G$ fixes $x$, and that if $g,h\in G$ both fix $x$, then $gh$ fixes $x$. This will establish that $G_x$ is a subgroup of $G$.

This won't finish the proof, although it will let you say that $\frac{\lvert G\rvert}{\lvert G_x\rvert}$ is the number of cosets of $G_x$ in $G$ (left or right, there's the same number either way). You still need to find a bijection between these cosets and the orbit of $x$ under $G$.