Homeomorphism between the unit disc and the unit square
Solution 1:
You do not mention whether you consider the closed or open unit disk resp. unit square. Here, let us consider the closed unit disk $D$ and the closed unit square $Q$.
Both receive their topologies as a subspaces of $\mathbb{R}^2$ which carries its standard topology. There are various equivalent descriptions of this topology. It is
1) the product topology on $\mathbb{R} \times \mathbb{R}$, where $\mathbb{R}$ has its standard topology.
2) the topology generated by the Euclidean norm $\lVert (x,y) \rVert_2 = \sqrt{x^2+y^2}$.
3) the topology generated by the maximum norm $\lVert (x,y) \rVert_\infty = \max(\lvert x \rvert, \lvert y \rvert)$.
Let us modify your definition of $f : \mathbb{R}^2 \to \mathbb{R}^2$ as follows: $$f(z) = \begin{cases} \frac{\lVert z \rVert_2}{\lVert z \rVert_\infty}z &\quad\text{if } z \ne 0 \\ \text{} 0 &\quad\text{if } z = 0 \\ \end{cases}$$ Your definition was $\frac{\lVert z \rVert_2^2}{\lVert z \rVert_\infty}z$ in the first line, but the present one is easier. To see that $f$ is continuous, let us use the maximum norm.
Continuity in $0$: $\lVert f(z) - f(0) \rVert_\infty = \lVert f(z)\rVert_\infty = \lVert z \rVert_2 \to 0$ as $z \to 0$.
Continuity in $w \ne 0$: $\lVert f(z) - f(w) \rVert_\infty = \left\lVert \frac{\lVert z \rVert_2}{\lVert z \rVert_\infty}z - \frac{\lVert w \rVert_2}{\lVert w \rVert_\infty}w \right\rVert_\infty = \left\lVert \frac{\lVert z \rVert_2}{\lVert z \rVert_\infty}(z-w) + (\frac{\lVert z \rVert_2}{\lVert z \rVert_\infty} - \frac{\lVert w \rVert_2}{\lVert w \rVert_\infty})w \right\rVert_\infty \le $ $\frac{\lVert z \rVert_2}{\lVert z \rVert_\infty}\lVert z - w \rVert_\infty + \left\lvert \frac{\lVert z \rVert_2}{\lVert z \rVert_\infty}\lVert w \rVert_\infty - \lVert w \rVert_2 \right\rvert \to 0$ as $z \to w$.
Moreover, $f(D) \subset Q$: We have $z \in D$ iff $\lVert z \rVert_2 \le 1$, thus $\lVert f(z )\rVert_\infty = \lVert z \rVert_2 \le 1$ which means $f(z) \in Q$.
Now define $g : \mathbb{R}^2 \to \mathbb{R}^2$ as follows: $$g(z) = \begin{cases} \frac{\lVert z \rVert_\infty}{\lVert z \rVert_2}z &\quad\text{if } z \ne 0 \\ \text{} 0 &\quad\text{if } z = 0 \\ \end{cases}$$ We can easily verify that $g$ is continuous and $g(Q) \subset D$. A final computation shows that $g \circ f = id$ and $f \circ g = id$.