What is the canonical definition of an open set?

In general topology there is no definition of what an open set is in terms of anything else. Instead, the open sets are precisely what constitutes the topology. So, a topological space is a pair $(X,\tau)$ where $X$ is a set and $\tau$ is a collection of subsets of $X$ that satisfy the axioms that: the intersection of any finite number of sets from $\tau$ is again in $\tau$ (in other words: $\tau$ is closed under finite intersection) and the union of any collection of elements from $\tau$ is again in $\tau$ (in other words, $\tau$ is closed under arbitrary unions). (Note that the empty intersection is $X$ and the empty union is $\emptyset$, so in particular $\emptyset$ and $X$ are elements in $\tau$).

This is the definition of a topological space. Now, in that context, elements of $\tau$ are said to be open sets. Any subset $Y\subseteq X$ that is not in $\tau$ is said to be not open. A subset $C\subseteq X$ is said to be closed precisely when $X-C$ is open, namely when $X-C\in \tau $.

So, in the context of a general topological space, there is no meaning to the question "what is the definition of the open sets", simply since there is no such definition. However, if you start with a metric space $(X,d)$, then you can define a set $A\subseteq X$ ($A$ may be empty!) to be open precisely when for all $a\in A$ there exists $\epsilon >0$ such that for all $x\in X$, if $d(x,a)<\epsilon$ then $x\in A$. One can then show that the collection of all open sets in a metric space forms a topology. So, with any metric space there is associated a canonical topology. This construction is the main reason for the terminology employed (at least in the first few pages) of most texts on topology.

The point is that on metric spaces where we have a notion of distance we can indeed make precise the notion of "all the points that are less than $\epsilon$ of some other point" using the concept of ball. An open set then is a set on which all points can be surrounded by some open ball completely contained in the set, so that you've made precise the notion of a set where each of it's points has all of it's neighbouring points yet in the set.

The point is that when you study metric spaces although you use a metric to define the open sets you soon see that many important results can be talked about only in terms of open sets. Those reuslts don't depend on the notion of distance itself, but rather, on the fact that we were able to formulate the idea of "sufficiently near a point", which is just the notion of openness.

So you think: "wait a moment, if I can generalize the notion of opennes to any set, then I'll be able to carry those results there" and then you start thinking on how you're gonna do this. What we do then is the same we do all the time in mathematics: we define something in a particular context, we see usefulness and we want to generalize, so we seek axioms that can be used as the base for a definition that will be far more general. One example that almost everyone is familiar with: this is what we do on linear algebra, we see linearity all arround, so we simply find the properties that grant linearity and promote them to axioms and define a linear space

It happens then that there are only three properties of open sets that are really fundamental and those properties are:

1. If you have a collection of even infinite open sets, then their union is still open.
2. If you have a collection of only finite open sets then their intersection is still open.
3. The set on which you're working with is open and so is the empty set.

All these properties are easily proved when you define open sets in metric spaces using balls. And as you see those properties are the ones that allows all your cool results on metric spaces you create the notion of a collection of sets with these properties:

Definition: Let $X$ be a set, a topology $\tau$ on $X$ is a collection of subsets of $X$ such that:

1. $\tau$ is closed under arbitrary unions.
2. $\tau$ is closed under finite intersections.
3. $X \in \tau$ and $\emptyset \in \tau$

Now you have a set and a topology on the set and you call the sets of the topology open sets of $X$ and you call $(X,\tau)$ a topological space. Now the important results of metric spaces you wanted to generalize will be possible to generalize in this framework where you postulate the open sets. It's not that you have a set and you find the open sets, it's just that if you present a collection of subsets of $X$ satisfying the properties $1$, $2$ and $3$ then those sets will be able to formulate one notion of openness for $X$.

Again, I'll compare with linear algebra because linear algebra is something that many people are comfortable with: you have a set $V$, then to make it a vector space and have the cool properties of linearity you just select two operations. If the axioms are satisfied you're ok.

It's the same here, it's you who has to present the open sets. In particular, if $(M,d)$ is a metric space, your best purposal is to say that the open sets are those such that every point has an open ball contained in the set, then you see that the properties hold and you're fine.

Edit: As pointed out in comment, what I wrote may lead someone to think that everything one can do on a metric space can also be done on a topological space. That's not true, as I've said: "you can transfer cool results from metric spaces", so indeed you cannot grant simply everything just with a topology, however, those things completly specified by relations using open sets (like the notion of compactness which can be stated in terms of open covers) can be brought to topological spaces.

One can define the topology that comes from a metric, and one can defined the topology that comes from a linear order, and one can define the topology that comes from any of a number of other sorts of structures, and in each case, one can check that the are topologies in the sense that unions of arbitrary sets of open sets are open and intersections of finite sets of open sets are open. (One can take the intersection of the empty set of open sets to be the whole space, and the union to be empty.)

But in general, an open set is a set that is open in some particular topology, and a topology on a set is any collection of subsets of a set that satisfies the definition. So a definition of the sort you seek cannot exist.

Let me give the analogous definitions for general topological spaces.

Given a non-empty set $X$, a topology on $X$ is a set $\mathcal T$ of subsets of $X$ such that:

(1) $\emptyset,X\in\mathcal T$,

(2) if $\mathcal A\subseteq\mathcal T$ then $\bigcup\mathcal A\in\mathcal T$, and

(3) if $\mathcal A$ is a finite, nonempty subset of $\mathcal T$ then $\bigcap\mathcal A\in\mathcal T.$

Now, a set $\mathcal B$ of subsets of $X$ is said to be a topological base on $X$ if:

(1) $\bigcup\mathcal B=X$ and

(2) for all $B_1,B_2\in\mathcal B$ and all $x\in B_1\cap B_2$ there is some $B_3\in\mathcal B$ such that $x\in B_3$ and $B_3\subseteq B_1\cap B_2$.

You should be able to show that if $\mathcal B$ is a topological base on $X$, then the set $$\mathcal T_{\mathcal B}:=\bigl\{\bigcup\mathcal A\mid \mathcal A\subseteq\mathcal B\bigr\}$$ is a topology on $X$. (In particular, it is the intersection of all topologues $$\mathcal T$ on $X$ such that $\mathcal B\subseteq\mathcal T$.) We call $\mathcal T_{\mathcal B}$ the topology on $X$ generated by $\mathcal B$.

Note that every topology is generated by a toological base (in particular, every topology is a topological base). A topological base generates a unique topology, but a given topology may be generated by many different topological bases.

Suppose we are given a topology $\mathcal T$ on $X$ and any topological basis $\mathcal B$ that generates $\mathcal T$. (For example: a metrizable topology, and the set of all open balls in that metric.) We say that a subset $A$ of $X$ is a neighborhood of a point $x\in X$ with respect to $\mathcal T$ if there is some $B\in\mathcal B$ such that $x\in B$ and $B\subseteq A$. A subset $A$ of $X$ is said to be open with respect to $\mathcal T$ if for each $x\in A$ we have that $A$ is a neighborhood of $x$ with respect to $\mathcal T$.

You should be able to show that $A$ is a neighborhood of $x\in X$ with respect to $\mathcal T$ if and only if there is some $U\in\mathcal T$ such that $x\in U$ and $U\subseteq A$. Likewise, $A$ is open with respect to $\mathcal T$ if and only if $A\in\mathcal T$.

There are various ways to express a topological structure; in certain situations, some methods are more useful than others.

One method that is frequently useful is to specify which sets you wish to consider "open neighborhoods". So there is a list of conditions this class of sets must satisfy to be adequate for defining a topology by being its class of open neighborhoods (we often call such a class a "basis" for the topology). Then, we learn how to construct everything else (e.g. which sets are open) in terms of this information.

However, it is also sometimes useful to specify a topology by listing which sets you wish to consider to be open. So there is a list of conditions this class of sets must satisfy to be adequate for defining a topology by being its class of open sets. Then, we learn how to construct everything else (e.g. which sets are open neighborhoods) in terms of this information.

Sometimes, you specify a topology by listing which sets should be the closed ones.

I think I've seen others, but they don't spring immediately to mind.

If we want to even define the notion of topological space, the typical approach is to choose one of these methods, and say that is the definition of what it means to be a topological space. Which method is a definition and which is not isn't all that important; in fact, you might even consider them all definitions of potentially different notions, until you get to the point where you prove the theorem that they are all essentially the same thing.