Why is continuity characterized with open sets?

I think what User Randall wrote in a comment is the main point: Only half of the emphasis in the definition of continuity as

The inverse images of all open sets are open

should lie on

The inverse images of all open sets are open

but at least half of it on

The inverse images of all open sets are open.

The intuition is that a set is open if around each point inside, there is still some wiggle room a.k.a. neighburhood around it. Granted that some open sets in a metric space also contain some points "far away", as in your example with the disjoint union of two balls -- but now that is where the all in the definition kicks in: To check continuity, you will also have to consider single balls. Really, really small single balls. All of them.

And in a metric space, it is clear that checking it for "very small" balls, small as in "your favourite $\epsilon$", suffices to prove it for all. Without a metric, it's harder to tell which open sets are small, so, well, let's just make the definition robust and demand it for all of them. (Actually, sometimes it suffices to check on various kinds of "basic" open sets.)

So the "all" is a placeholder for arbitrarily small, which technically does not make sense in a general topological space. As soon as it does -- in a metric space --, you can replace "all" by "arbitrarily small", and making that rigorous will give the usual $\epsilon-\delta$-definition back.

I personally have always had problem with the definition of continuity with respect to open sets. And my issue is not that open sets are too general, but my issue is that the definition looks sort of twisted and counter-intuitive to what one would expect it to be.

If $f: X\to Y$ is a mapping between two topological spaces $X$ and $Y$, then $f$ is continuous if and only if $f^{-1}(U)$ is an open subset of $X$ for any $U$ that is open is $Y$. In metric spaces, we know that a set is open if we can find a ball of small enough radius centered around each of its points. So, one can easily see that this definition does imply the good old $\epsilon-\delta$ definition of continuity that we were taught in calculus.

But still, the idea of working with inverse images looks mysterious to me. A more natural looking definition, which is equivalent, is the definition that uses the idea of closures in topology. $f$ is continuous if and only if

$$f(\mathrm{cl}(S)) \subseteq\mathrm{cl}(f(S))$$

is true for any set $S \subseteq X$. First of all, it looks more natural because we're working with $f$ and not $f^{-1}$. Secondly, a closure of a set $S$ is the set of all points that are arbitrarily "close" to it and therefore, this statement says that $f$ is continuous if and only if it sends close points to an arbitrary set $S$, to points that are close to its image.

If you are in a good topological space like a metric space, you can define the closure of a set using the idea of the limit of a sequence. A point is in the closure if there exists a sequence that approaches toward it. Then $f$ can be equivalently defined as being continuous if and only if

$$\lim_{n\to\infty}f(a_n)=f(\lim_{n\to\infty}a_n)$$

This is also more familiar and in essence is very similar to saying that $f$ maps nearby points to nearby points.

Now, there are at least four axiomatic ways to define abstract topology: using the properties of open sets, using the properties of closed sets, using the properties of neighborhoods and using the properties of the closure operator which is due to Kuratowski if I'm not mistaken. It can be very instructive to check that all of these axiomatic systems turn out to be equivalent and see how continuity is defined in each of them.

Metric spaces in analysis are a special case of the first system of axioms for topology (open sets) where our space has a basis of open balls, because the definition of an open ball is very straightforward when we have a metric on $X$.

It appears that most of your confusion comes from your notion of continuity being tightly coupled to the notion of a metric space. But in fact the notion of continuity for metric spaces is tremendously natural, and it is not necessary to define concepts like open balls or open sets to get at continuous functions over metric spaces.

However for functions between non-metric spaces, one cannot simply decide about what being "close" or "near" means, or even define what a "ball" is. It is in these cases that topologies and open sets shine.

One should note that there isn't anything special about open sets vs closed sets. It is perfectly reasonable to define topologies and continuity in terms of closed sets. (And in fact it is sometimes easier, such as in algebraic geometry).

Relatedly, you also seem to conflate the idea of two points being close and two points being contained in a single open set (in particular, you seem to dislike it when this open set is formed as a union of disjoint open balls in a metric space). But in fact any two points in any topological space are contained in an open set: namely the whole space (which is open). The intuition that being in an open set somehow means that two elements are close is a very dull intuition, and should be sharpened.

It is somehow better to think of two points as being close if they are contained in "lots" of open sets. And still this intuition leaves room for more careful examination.

You seem to have a misconception about what the notion of openness is supposed to represent. You write:

I know that open sets are the set of points under some topology that are close. i.e. we only need sets to classify what points are considered close. Which seems to me the main motivation why open sets were chosen, but the fact that disjoint open ball pass the test and are considered close by particularly disturbs me for some reason. Why is this specific complaint ok to ignore? What justifies not being worried about it?

It's not true that an open set is a set of points that are "close together". Open sets do not, and are not supposed to, be sets of points that are "all close to each other". That's why it's fine a union of disjoint open balls is considered an open set: contrary to what you write here, disjoint open balls are not "considered close by".

It looks like this is at the core of your confusion. Maybe things will make more sense now?

I'm not sure why open balls are any better than open sets at capturing the notion of "closeness". If a ball has radius 1000000, the points in it won't be close in any sense. What you seem to be thinking of is not a particular open ball, but rather the set of all open balls around a point $p$. This more accurately captures the $\epsilon-\delta$ definition, as in that definition you are allowed to make $\delta$ as small as you like. Thus you may choose the open ball so that the points in it are arbitrarily near to $p$. Now, you can also do this for open sets, since every open ball is an open set.

This is formalized by the idea of a neighborhood basis, defined as any collection $N_p$ of open neighborhoods of $p$ which are ``arbitrarily small'' in the sense that if $U$ is any open set containing $p$, then there is a set in $N_p$ which is a subset of $U$. If the topological space is a metric space, we can take $N_p$ to be the set of all balls centered at $p$.

By making the neighborhoods become smaller and smaller, we can recover the idea of closeness encapsulated by the $\epsilon-\delta$ definition. To be precise, suppose $X$ and $Y$ are metric spaces, and let $p$ be a point of $X$. Choose an arbitrarily small element of the neighborhood basis of $f(p)$, that is, a ball $B_\epsilon(f(p))$ for arbitrarily small $\epsilon$. If $f$ is continuous in the topological sense, then $f^{-1}(B_\epsilon(f(p))$ is open, and it contains $p$, so it must contain some open ball $B_\delta(p)$. To say that $f(B_\delta(p))\subseteq B_\epsilon(f(p))$ means that if $|p_1-p|<\delta$, then $|f(p_1)-f(p)|<\epsilon$, which is the $\epsilon-\delta$ definition of continuity.

This shows that we can recover the $\epsilon-\delta$ definition by only applying the topological definition to those open subsets of $Y$ which happen to be open balls. Conversely, suppose we know that $f^{-1}(B_\epsilon(q))$ is open for all balls $B_\epsilon(q)$ in $Y$. Then it must follow that $f^{-1}(V)$ is open for every open set $V$. To see this, write $V=\bigcup_\alpha B_\alpha$ as a union of balls $B_\alpha$ (which we can do since $V$ is open). Then $f^{-1}(V)=\bigcup_\alpha f^{-1}(B_\alpha)$. By hypothesis, each $f^{-1}(B_\alpha)$ is open, therefore $f^{-1}(V)$ is also open.

Thus we could give an alternative characterization of continuity by simply requiring that $f^{-1}(V)$ be open when $V$ is an element of the neighborhood basis. However, if for $q\in Y$ we take $N_q$ to be the set of all open sets containing $q$, this is also a neighborhood basis, and so we see that this isn't so different from the usual definition.

I will remark that for a metric space, there is also a countable neighborhood basis of any point $p$, obtained by considering only those balls $B_\delta(p)$ where $\delta\in\mathbb Q^+$. This is essentially what allows us to express continuity in terms of limits, as in stressed out's answer.

Finally, let me address the issue of disjoint open sets. Suppose $V$ is an open neighborhood of $f(p)$ in $Y$ such that $f^{-1}(V)$ is a union of two disjoint open sets $U_1$ and $U_2$ in $X$, where, say, $p\in U_1$. If we consider $U_1$ to be the "nearby" points, all this says is that there are also some "non-nearby" points (in $U_2$) which map into $V$, which is not a problem.