Domain of an operator in functional analysis
Solution 1:
This is an admittedly confusing abuse of terminology which your book appears to make even more confusing by using somewhat nonstandard terminology. The following is more standard but also confusing in its own way. For what follows let $X$ and $Y$ be Banach spaces.
A bounded linear operator from $X$ to $Y$ is a function $T: X \to Y$ which is linear and norm continuous.
An unbounded linear operator from $X$ to $Y$ is a pair $(T,D_T)$ where $D_T$ is a linear subspace of $X$ and $T: D_T \to Y$ is a linear map.
Comments:
An unbounded linear operator from $X$ to $Y$ is not in general a function from $X$ to $Y$, and the definition does not claim it is. The phrase "from $X$ to $Y$" is part of what is being defined.
Because unbounded operators are not functions, extra care must be taken: two unbounded linear operators from $X$ to $Y$ cannot necessarily be added, and given unbounded linear operators from $X$ to $Y$ and from $Y$ to $Z$ they do not necessarily have a composition.
"Unbounded" really means "not necessarily bounded"; every bounded linear operator is an unbounded linear operator.
Some authors require that the domain $D_T$ of $T$ is a dense subspace of $X$. With this convention one can conceive of operators which are neither bounded nor unbounded. When this is not part of the definition, the assumption is added via the phrase "densely defined".
Adjoints can be very confusing: there are densely defined unbounded operators such that the largest possible domain of definition for the adjoint is $\{0\}$.
The only real way to sort through all of this language is to look at some examples and results. The theory was invented by Von Neumann in the 20's to make precise sense of what physicists were doing in quantum mechanics. They had stumbled onto the idea that classical notions like position and momentum can be profitably viewed as linear operators on Hilbert space which satisfy certain relations. Mathematicians noticed, however, that there are no bounded operators on Hilbert space which satisfy these relations, and in particular the physicists' use of spectral theory had no mathematical basis. So Von Neumann invented the language of unbounded operators to make sense of the operators that physicists were doing. In particular he generalized the spectral theorem to certain classes of unbounded operators.
Maybe it would help to look at a quick example. Consider the operator $D = i \frac{d}{dx}$, a linear map $C_{per}^\infty[0,1] \to C_{per}^\infty[0,1]$, the space of smooth periodic functions on the interval $[0,1]$. Notice that $D$ has a rich supply of eigenfunctions: setting $e_n(x) = e^{-2 \pi i n x}$, we find that $e_n$ is an eigenfunction of $D$ with eigenvalue $2 \pi n$. The set of functions $\{e_n\}$ forms an orthonormal basis for the Hilbert space $L^2[0,1]$, so even though $D$ isn't an operator on $L^2[0,1]$ we can in a sense diagonalize it and use Hilbert space techniques to study it.
By contrast, consider the same operator $D = i \frac{d}{dx}$, this time viewed as a linear map $C_0^\infty(0,1) \to C_0^\infty(0,1)$, the space of smooth functions on the interval $[0,1]$ which vanish at $0$ and $1$. Now the only eigenfunctions for $D$ are constant functions, so it is not clear how one would diagonalize $D$. Secretly we know that we should just enlarge the domain of $D$ to include all periodic functions, but this isn't immediately obvious. Part of Von Neumann's accomplishment was to understand abstractly why $D$ with the domain $C_{per}^\infty[0,1]$ is "good" but $D$ with the domain $C_0^\infty(0,1)$ is not.
Solution 2:
A linear operator $T$ is actually a pair $(D_T,T)$, where $D_T$ is a subspace of $X$ and $T\colon D_T\to Y$ is a linear map. So two linear operators $S$ and $T$ are considered to be equal if they have the same domain $D$, and $Sx=Tx$ for all $x\in D$.
This allows to consider the concept of extension: $(D_T,T)$ is an extension of $(D_S,S)$ if $D_S\subset D_T$ and for all $x\in D_S, Sx=Tx$.