Why don't we take clopen maps as morphisms of the category of topological spaces?

Here, by clopen maps I mean a function mapping an open set into an open set and a closed set into a closed set. We say continuous maps are the morphisms of the category of the topological spaces. But isn't it more natural to consider clopen maps instead of continuous maps? For example, we say group homomorphisms are the morphisms, and a group homomorphism $h:G\rightarrow H$ preserves the group structure of "$G$" in "$H$". On the other hand, in some sense, a continuous map $f:X\rightarrow Y$ preserve the topological structure of "$Y$" in "$X$". The direction is reversed. I am wondering if there is any good explanation for this. Thanks!

Well, the obvious reason is that such a definition would lack the properties of continuous maps that we'd expect. For example, for the Sierpiński space consisting of $\{\bot,\top\}$ with open sets $\{\{\},\{\top\},\{\bot,\top\}\}$ the only "clopen map" is the identity function. If we defined continuous map as "clopen map" then, in this case, not even constant functions would be continuous. Still, this tells you that this is a bad definition for "continuous", but not how you might be led to the right one, particularly without a moment of insight.

We can be more systematic though. For an algebraic structure, like a group, the structure consists of a collection of operations operating on a set. For example, inversion is a unary group operation $G \to G$. Given an operation $\mathtt{i}_G:G\to G$ and a function $\varphi:G\to H$ which, let's say, is a bijection, then a very natural notion of "preserving" $\mathtt{i}$ is that $\mathtt{i}_G(g) = \varphi^{-1}(\mathtt{i}_H(\varphi(g)))$ which clearly leads to $\varphi(\mathtt{i}_G(g)) = \mathtt{i}_H(\varphi(g))$ which doesn't require $\varphi$ to be a bijection.

The structure for a topological space $X$ is topology $\mathcal{T}_X$ which is a subset of $\mathcal{P}(X)$, or, equivalently, a subset of $X\to\mathbf{2}$. We could take the view that each open subset $U_X\in\mathcal{T}_X$ is an "operation", $X \to \mathbf{2}$. The natural notion is, given a function $f : X \to Y$ is that an operation is preserved via $U_X(x) = U_Y(f(x))$ for a suitable $U_Y\in\mathcal{T}_Y$. In particular, this states that given an open subset of $Y$, $U_Y$, we get an open subset of $X$ via $U_Y \circ f$. Viewing $U_X$ and $U_Y$ as subsets, the preservation condition is exactly that $f^{-1}(U_Y) = U_X$. Of course, this is different from the algebraic case as we don't have a fixed set of operations that are rigidly mapped one to the other. At any rate, the "backwardness" isn't so surprising when you consider the contravariance of $\mathcal{P}(X)$ (i.e. $X \to \mathbf{2}$) as a functor in $X$.

Of course, this doesn't explain what operations topologies should be closed under.

The description of continuity in terms of preimages of open sets/closed sets is primarily of computational convenience. A more intuitive definition is that a function $X\xrightarrow{f}Y$ is continuous if it maintains closeness of points to sets in the sense that $f(\overline U)\subseteq\overline{f(U)}$ for any $U\subseteq X\xrightarrow{f}Y$. Here $\overline U$ is the closure of a subset $U\subseteq X$ and consists of all points of $X$ that are close to $U$, i.e. a topology on a set $X$ can be interpreted as the specification of a (well-behaved) closeness relation between points of $X$ and subsets of $X$, and continuity as the preservation of this closeness relation.

With a little bit of thought, it should be easy to see that the notions of closed and open maps, while preserving closed or open subsets, do not necessarily preserve the closedness relation; conversely, maintaining the closedness relation does not mean that closed sets or open sets are preserved. So to answer your question, we don't use clopen maps because they preserve a different structure from the one we are trying to capture. (In fact, I think in this context the notions of open and closed are somewhat artificial, perhaps merely artifacts of the attempt to describe the closedness relation).

To address the comparison to groups, not all structures are given by operations, structures can also be given by relations.

Because then $f : \mathbb{R} \to \mathbb{R}$, $x \mapsto e^x$ would not be a morphism (its image is not closed). Recall that almost every notion in mathematics is motivated by examples, and this includes the definition of a category and explicit examples of categories. You don't want to just play around with the axioms, you want to study the examples you are actually interested in. And a definition of a morphism of spaces which does not include the exponential function is certainly not natural.

Another approach to topological spaces is provided by Kuratowski spaces. Here, one has a relation $x \prec A$ between points $x$ and subsets $A$ of the underlying set, which is supposed to mean that $x$ lies in the closure of $A$. This relation $\prec$ has to satisfy some axioms. Then, a map $f$ is continuous if and only if it preserves this relation $\prec$: $$x \prec A \Rightarrow f(x) \prec f(A)$$

You might also want to have a look at frames; here the morphisms are really algebraic homomorphisms as you suggest. Frames resp. locales are a generalization of topological spaces. This is also called "pointless topology".

We construct a category Top whose arrows are the continuous functions because we are interested in studying continuous functions.

The structuralist philosophy is that "the structure of being a topological space" is precisely the same thing as "being an object of Top".

We can recover the usual description of a topological space (up to natural bijection of sets) from the category structure of Top. Let $1$ be the one-point set, $S$ be the Sierpinski space, and $u : 1 \to S$ be the continuous whose image is the open point of $S$.

• The set of points of an object $X$ is (naturally isomorphic to) $\hom(1, X)$
• The set of open sets of an object $X$ is (naturally isomorphic to) $\hom(X, S)$
• A point $f : 1 \to X$ lies in an open set $g : X \to S$ iff $g \circ f = u$

(note that $1$, $S$, and $u$, can also be identified from the category structure)

So, the "set of points" is a covariant functor and the "set of open sets" is a contravariant functor — we really should expect a 'structure preserving map' to act backwards on the open sets.

It is of interest to note that $\mathbf{Set}^\text{op}$, the opposite category to sets, is equivalent to the category of complete atomic boolean algebras (and join/meet preserving lattice homomorphisms) — the equivalence is given by sending a set $X$ with its power set $\mathcal{P}(X)$.

Since a lot of our intuition about objects comes from sets, this suggests a general principle

• The 'covariant structure' on objects looks like operations on points
• The 'contravariant structure' on objects looks like operations on subsets

Your examples from universal algebra are all expressed in terms of operations on points, which is why you only see homomorphisms being required to preserve the covariant structure.

For an algebraic example with important contravariant structure, there is the category of local rings and local homomorphisms.

In this category, the unique maximal ideal is an important part of the structure of an object — and, as one might expect, a ring homomorphism $f : R \to S$ is a local homomorphism if and only if $f^{-1}(\mathfrak{m}_S) = \mathfrak{m}_R$.

(Note that the definition is often presented in the equivalent form $f(\mathfrak{m}_R) \subseteq \mathfrak{m}_S$, but I find that much less intuitive and harder to remember)