Shrinking Group Actions

Solution 1:

I don't know if my answer will be satisfactory, but it is actually very easy.

A continuous bijection $\varphi$ is a homeomorphism iff it is open and/or closed (openness/closeness of $\varphi$ rephrases continuity of the inverse $\varphi^{-1}$). As written in the question if $Y$ is compact (enough if $H\setminus Y$ is) and $G\setminus X$ is Hausdorff, then $\theta$ is obviously closed.

On the other hand, when $Y$ is open, then it easilly follows that the map $\theta$ is open.

Proof. Denote the quotient map $Y\to H\setminus Y$ by $\pi^Y$ and similarly let $\pi^X:X\to G\setminus X$ be the other obvious projection. We want to check that for all open $U\subseteq H\setminus Y$ the image $\theta(U)$ is open in $G\setminus X$. Since $U=\pi^Y\left({(\pi^Y)}^{-1}(U)\right)$ and $\theta\circ\pi^Y = \pi^X$, we can write $\theta(U)=\pi^X\left({(\pi^Y)}^{-1}(U)\right)$ and it suffices to prove that $\pi^X$ is open.

Consider any open $V\subseteq X$. We need to check if $\pi^X(V)$ is open in the quotient, so by definition of the quotient topology if its inverse image via the quotient map is open in $X$. We have the equality $${(\pi^X)}^{-1}\Big(\pi^X(V)\Big)= G.V \overset{def.}{=} \{g.v\ |\ g\in G,\ v\in V \}$$ and $G.V$ is open as the sum of open sets of the form $g.V$. Thus $\pi^X$ is open and so is $\theta$, which ends the proof.

Summing up: it is very common that a bijective $\theta$ is a homeo: it is enough to assume that $Y$ is compact (and $G\setminus X$ Hausdorff) or open (and nothing more).

From the above proof we can see that a sufficient and necessary condition for openness of $\theta$ (equivalently: for it to be a homeomorphism) is that for any open subset $W\subseteq Y$ (enough to check for W that are $H$-stable) the set $G.W$ is open in $X$.