What are relative open sets?

I came across the following:

Definition 15. Let $X$ be a subset of $\mathbb{R}$. A subset $O \subset X$ is said to be open in $X$ (or relatively open in $X$) if for each $x \in O$, there exists $\epsilon = \epsilon(x) > 0$ such that $N_\epsilon (x) \cap X \subset O$.

What is $\epsilon$ and $N_\epsilon (x) $? Or more general, what are relatively open sets?

Forget your definition above. The general notion is:

Let $X$ be a topological space, $A\subset X$ any subset. A set $U_A$ is relatively open in $A$ if there is an open set $U$ in $X$ such that $U_A=U\cap A$.

I think that in your definition $N_\epsilon(x)$ is meant to denote an open neighborhood of radius $\epsilon$ of $x$, ie $(x-\epsilon,\ x+\epsilon)$. As you can see, this would agree with the definition I gave you above.

Recall that generally, $O$ is open if for every $x\in O$ there exists some $\varepsilon>0$ such that $N_\varepsilon(x)\subseteq O$.

Being open relative to $X$ means that there is some open set $O'$ such that $O=O'\cap X$, and equivalently for every $x\in O$ there is some $\varepsilon>0$ such that $N_\varepsilon(x)\cap X\subseteq O$.

For example $O=\{0\}$ is not open in $\mathbb R$, but if we consider $X=\{0\}$ then for $\varepsilon=1$ we have that $N_\varepsilon(0)\cap X\subseteq O$, and therefore it is open relative to $X$.