A continuous group action is an action by homeomorphisms

This has nothing to do with spaces - it's entirely a consequence of the definition of a group action on a set.

An action of $G$ on $X$ has to play well with the structure of $G$: writing "$g\cdot x$" for the action of the group element $g\in G$ on the set element $x\in X$, we have to have $$(gh)\cdot x=g\cdot (h\cdot x)$$ for all $g,h\in G$ and all $x\in X$, and moreover $e\cdot x=x$ for all $x\in X$ when $e$ is the identity of $X$. This automatically means that each map $x\mapsto g\cdot x$ must be a bijection, since groups have inverses.

A monoid action need not "consist of bijections," but a group action must.

When a group, any group, acts on a set, any set, the action of any group element is a bijection of the set. This is so because the composition of the action of two elements is the action of their product, and every group element is invertible. Therefore, you should always be able to undo the action and return to the state where nothing moves (which is the action of the identity element).