Homeomorphisms of X form a topological group

Here you don't need much topology, it boils down to pure manipulation of sets and bijective functions, and the two following facts: $(*)$ closed subsets of compact spaces are compact, and $(**)$ compact subsets of Hausdorff spaces are closed. Just take complements and use the fact that $h$ is by definition a bijection, so $$\begin{array}{RCL} h\in S(C,U) & \Longleftrightarrow & h(C)\subset U\\ & \Longleftrightarrow & X\setminus U\subset \underbrace{X\setminus h(C)}_{=h(X\setminus C)}\\ & \Longleftrightarrow & h^{-1}(X\setminus U)\subset X\setminus C \\ & \Longleftrightarrow & h^{-1}\in S(X\setminus U,X\setminus C) \end{array}$$ (which is a subbasic open neighborhood) i.e. $$\mathrm{inv}^{-1}( S(C,U))=\mathrm{inv}(S(C,U))=S(X\setminus U,X\setminus C)$$ which proves the inversion map $\mathrm{inv}$ to be continuous.