$g$ a restriction of a homeomorphic function $f$, $g$ also homeomorphic?
Solution 1:
In the part about continuity, note that $U\subseteq B$ open in $B$ means only that $U=B\cap U'$, where $U'$ is open in $Y$. Then $$g^{-1}(U)=f^{-1}(U)\cap A=f^{-1}(U')\cap f^{-1}(B)\cap A=f^{-1}(U')\cap A$$ so it is an open subset of $A$. This equation works as long as $f(A)⊆B$.
Or show the continuity of $g$ by the following diagram
$$\begin{array}{ccc}
\ A & \xrightarrow{i} & X\\
g\downarrow & & \ \downarrow f\\
\ B & \xrightarrow{j} & Y
\end{array}$$
By the universal property, $g$ is continuous if $jg$ is, but that's the same as $fi$ which is a composition of continuous functions.
You could also try to find the inverse of $g$. If $\bar f:Y→X$ denotes the inverse of $f$, then its restriction $\bar g$ to $B$ is a map $\bar g:B\to A$ since $\bar f(B)=f^{-1}(B)=f^{-1}(f(A))=A$. Now $$\bar gg(a)=\bar ff(a)=a\qquad g\bar g(b)=f\bar f(b)=b$$ So $g $ and $\bar g$ are inverse to each other.