Are the rationals a closed or open set in $\mathbb{R}$?

I have a feeling they are neither closed nor open as the $\mathbb{R} \setminus \mathbb{Q}$ cannot be open or closed either...

In the usual topology of $\mathbb{R}$, $\mathbb{Q}$ is neither open nor closed.

The interior of $\mathbb{Q}$ is empty (any nonempty interval contains irrationals, so no nonempty open set can be contained in $\mathbb{Q}$). Since $\mathbb{Q}$ does not equal its interior, $\mathbb{Q}$ is not open.

The closure of $\mathbb{Q}$ is all of $\mathbb{R}$: every real number is the limit of a sequence of rationals, so every real number lies in the closure of $\mathbb{Q}$. Since $\mathbb{Q}$ does not equal its closure, it is not closed.

Naturally, since $\mathbb{Q}$ is not open, its complement is not closed; since $\mathbb{Q}$ is not closed, its complement is not open.

But this is in the usual topology. $\mathbb{R}$ can be endowed with lots of topologies, and it is certainly possible for $\mathbb{Q}$ to be open (or closed) in some of them. For example, in the discrete topology, where every subset of $\mathbb{R}$ is both open and closed, $\mathbb{Q}$ is both open and closed.

If the rationals were an open set, then each rational would be in some open interval containing only rationals. Therefore $\mathbb{Q}$ is not open.

If $\mathbb{Q}$ were closed, then its complement would be open. Then each irrational number would be in some interval containing only irrational numbers. That doesn't happen either. This one is harder to prove: showing that every interval on the line contains some rationals depends on the fact that the ordered field $\mathbb{R}$ is Archimedean.

Neither. Their interior is empty and their closure is the entire line.

They are neither open nor closed. Note that they cannot be closed since the closure of $\mathbb{Q}$ is $\mathbb{R}\ne \mathbb{Q}$, but they cannot be open either since the same holds for their complement (as you pointed out).

Remember that in metric spaces (those with a notion of distance), we can check if a set $A$ is closed by checking that the limits of elements in $A$ rest in $A$ (i.e., for any convergent sequence $(a_i)_{i\in\mathbb{N}}$, such that $a_i\in A$, then $\lim_{i\to +\infty} a_i \in A$ ).

$\mathbb{R}$ with the usual distance ($d(x,y)=|x-y|$) is probably the typical example of a metric space.

With that in mind, if $\mathbb{Q}$ were closed, then any sequence of rationals would converge in the rationals, which we know is false; for example:

$$(1+1/n)^n \to e.$$

On the other hand, if $\mathbb{Q}$ were open, then its complement ($\mathbb{I}$, the irrationals) would be closed, which again we can easily see that it's not. (E.g., take any sequence of rationals that converges to an irrational and subtract from each element in the sequence this irrational: where will it converge?)

But even though $\mathbb{Q}$ is neither open nor closed, it is not that complicated topologically speaking. $\mathbb{Q}$ is what we call an $F_\sigma$ set ($F$ from fermé (closed in French), and $\sigma$ from sum (meaning union)). These are sets that are countable unions of closed sets. A bunch of natural examples of sets are $F_\sigma$, such as the points of discontinuity of a function. The "dual" class is known as $G_\delta$, the countable intersections of open sets.