Solution 1:

Expounding on the comments:

An $n \times m$ matrix is by definition a function $A: \{1, \ldots, m\} \times \{1, \ldots, n\} \to \mathbb{R}$. Most of the time we write $A_{ij}$ for its value at $(i,j)$ instead of $A(i,j)$. If you think about the definition of a vector, this is equivalent to looking at points of $\mathbb{R}^{mn}$. So for $n \times n$ square matrices, they are just points of $\mathbb{R}^{n^2}$.

On $\mathbb{R}^k$ we have a bunch of norms. It's a basic theorem of functional analysis that these are all equivalent and define equivalent topologies (this is essentially what the point of equivalence is). Perhaps the simplest is just the $\ell_1$-norm: the sum of the absolute values of the entries of the matrix.

So now we can talk about distance and other similar notions like compactness. For example, we have the function $\det: \mathbb{R}^{n^2} \to \mathbb{R}$. By its definition, it is a polynomial in the entries of the matrix. Hence it is a very nice smooth function. The group $\text{GL}(n,\mathbb{R})$ is

$$ \text{GL}(n, \mathbb{R}) = \{ A \in \mathbb{R}^{n^2} \mid \det(A) \neq 0 \}$$

and so is the preimage of an open set under a continuous function. Therefore it is an open set.

The special linear group consists of the matrices with determinant $\pm 1$, and so similarly forms a closed subgroup.

For a compact example, consider the orthogonal matrices. An orthogonal matrix, if you think about it, is one that satisfies $n^2$ linear equations in its entries. So they'll form a closed set (a subgroup, even). Moreover, we can see that they are actually bounded (the column sums cannot exceed $1$), and so by the Heine-Borel theorem AKA the characterisation of compactness in $\mathbb{R}^k$ we know that they are a compact subgroup.

A nice application of this topological nonsense is the Cayley-Hamilton theorem (any matrix is a root of its characteristic polynomial). Working over $\mathbb{C}$, one can show that the diagonalizable matrices are in fact dense in $\mathbb{C}^{n^2}$. The Cayley-Hamilton theorem is true for these, and hence extends to all complex matrices. Moreover, one can show by some algebraic nonsense that this implies it's true for all commutative rings (with unit).

Solution 2:

I give this answer to expand on the comments of Jyrki Lahtonen and Abramo.

Let $M = gl(n, \mathbb{R})$ which is the set of $n \times n$ matrices over $\mathbb{R}$. A global coordinate chart $x_{ij}$ is given by the component mapping: $$ x_{ij}(A) = e_i^TAe_j $$ where $e_i$ serves as the standard basis of $\mathbb{R}^n$. It is easy to show this is a bijection from $M$ to $\mathbb{R}^{n^2}$ and it gives $M$ the structure of an $n^2$-dimensional manifold.

The mapping $det: M \rightarrow \mathbb{R}$ has $GL(n, \mathbb{R})$ as the complement of the fiber over $0$ of this map; $GL(n, \mathbb{R}) = M-det^{-1} \{ 0 \}$. As such it is simply a restriction of $M$ to a subset. I leave it to you to ascertain if $GL(n, \mathbb{R})$ so constructed is open or closed. I'll give you a hint: the determinant map is formed by polynomials of coordinate maps on $M$ hence is clearly continuous and $\{0 \}$ is a closed set.