Relationship among the function spaces $C_c^\infty(\Omega)$, $C_c^\infty(\overline{\Omega})$ and $C_c^\infty(\Bbb{R}^d)$
Solution 1:
Since it seems that the author of the book doesn't explicitly define what it means for a function defined on a closed set to be differentiable, I can only guess that for him differentiable functions on a closed set are restrictions of differentiable functions defined on a larger, open set and so
$$ \mathcal{D}(\overline{\Omega}) = \{ f \colon \overline{\Omega} \rightarrow \mathbb{R} \, | \, \exists g \in C^{\infty}(\mathbb{R}^n) \text{ s.t } f = g|_{\overline{\Omega}}, \mathrm{supp}_{\overline{\Omega}} f \subset \subset \overline{\Omega} \}. $$
Using this definition, we have $\mathcal{D}(\Omega) \subset \mathcal{D}(\overline{\Omega}) \subset C^{\infty}(\Omega)$. To see the difference between $\mathcal{D}(\Omega)$ and $\mathcal{D}(\overline{\Omega})$, take $\Omega = (0,1)$ and $g \colon \mathbb{R} \rightarrow \mathbb{R}$ smooth with compact support such that $g(x) = 1$ for $x \in [0,1]$. Then $g \notin \mathcal{D}((0,1))$ but $g \in \mathcal{D}([0,1])$. More generally, one can characterize $\mathcal{D}(\overline{\Omega})$ as
$$ \mathcal{D}(\overline{\Omega}) = \{ f \colon \Omega \rightarrow \mathbb{R} \, | \, \exists g \in C^{\infty}(\mathbb{R}^n) \text{ s.t } f = g|_{\Omega}, \,\, \mathrm{supp}_{\mathbb{R}^n} g \subset \subset \mathbb{R}^n \} $$
so that functions in $\mathcal{D}(\overline{\Omega})$ are restrictions to $\Omega$ of functions in $\mathcal{D}(\mathbb{R}^n)$. I've found your notation $C^{\infty}_c(\overline{\Omega})$ used together with the latter definition in Elliptic Problems in Nonsmooth Domains (page 24) by Pierre Grisvard.
If $\Omega = \mathbb{R}^n$, then $\mathcal{D}(\Omega) = \mathcal{D}(\overline{\Omega})$ as the notation suggests.