Why do the $n \times n$ non-singular matrices form an "open" set?
Solution 1:
"Open" is in the sense of topology. A set can be endowed with an extra structure, called a topology, that is a decision about which subsets of our set will be called "open" (the decision cannot be arbitrary however, there are some rules). The real numbers, $\mathbb{R}$, can be given a topology, called the "usual topology", where we call $S\subseteq \mathbb{R}$ "open" when, for any $x\in S$, there is an open interval $(a,b)$ such that $x\in (a,b)$ and $(a,b)\subseteq S$.
The set $M_n(\mathbb{R})$ of all $n\times n$ real matrices can be identified with $\mathbb{R}^{n^2}$, via $$\begin{pmatrix}a_{11} & \cdots & a_{1n}\\ \vdots& \ddots & \vdots\\ a_{n1}& \cdots& a_{nn}\end{pmatrix}\mapsto(a_{11},a_{12},\ldots,a_{1n},a_{21},\ldots,a_{nn})$$ and because $\mathbb{R}$ has its usual topology, we can put the product topology on $\mathbb{R}^{n^2}$, and hence on $M_n(\mathbb{R})$.
The determinant function $\det:M_n(\mathbb{R})\to\mathbb{R}$ sends an $n\times n$ matrix to its determinant. It is continuous, because it is, in fact, a polynomial in the entries of the matrix (i.e., under our identification, it is a polynomial function in $n^2$ variables from $\mathbb{R}^{n^2}$ to $\mathbb{R}$). One of the properties of continuous functions (this is in fact what the definition of a continuous function is in general) is that the preimage of an open set is open. Then because a matrix is non-singular if and only if its determinant is not $0$, we have that $$\text{GL}_n(\mathbb{R})=\{A\in M_n(\mathbb{R})\mid A\text{ is non-singular}\}={\det}^{-1}(\mathbb{R}-\{0\})$$ and $\mathbb{R}-\{0\}$ is an open set of $\mathbb{R}$, hence $\text{GL}_n(\mathbb{R})$ is an open subset of $M_n(\mathbb{R})$.
The notation $\text{GL}$ comes from the fact that this is often called the general linear group.
It's also interesting to contrast this result with another result, which I showed in a different answer, that the set of $m\times n$ matrices having rank less than or equal to $k$, for some $k\leq\min\{m,n\}$, is a closed subset of $M_{m\times n}(\mathbb{R})$. Note that a square $n\times n$ matrix is non-singular if and only if it has full rank, i.e. its rank is equal to $n$.
Solution 2:
Somewhat informally: if you take a matrix, and change the entries of it a little bit, then the determinant also changes by a little bit. (Formally, the determinant is a continuous function of the matrix entries.)
So if you start with a matrix with non-zero determinant, and change its entries a little bit, the result will also have non-zero determinant. So matrices that are "near" a non-singular matrix are themselves non-singular; that's what it means for that the set of non-singular matrices is an open set.
In contrast, the set of singular matrices is not open. Since it's the complement of an open set, it's called a closed set: Wikipedia article on closed sets. It's not open because if you take a non-singular matrix and change it a little bit, it (usually) won't be non-singular any more; it's closed because if you take the limit of a sequence of singular matrices (in the "obvious" componentwise way) and that limit exists, it is a singular matrix.
(All this is made more formal in Zev Chonoles' answer.)