Spectral Measures: Square Root

Let $S$ be any densely-defined selfadjint linear operator on a Hilbert Space with spectral measure $E$. The range of $E[-\lambda,\lambda]$ is in the domain of every positive power of $S$. In fact, $$ \mathscr{C}=\bigcap_{n=1}^{\infty}\mathcal{D}(S^{n}) $$ is a core for $S$. This is because the range of $E[-\lambda,\lambda]$ is in $\mathscr{C}$, and $x\in\mathcal{D}(S)$ implies $$ \lim_{\lambda\uparrow\infty}E[-\lambda,\lambda]x = x,\\ \lim_{\lambda\uparrow\infty}SE[-\lambda,\lambda]x = Sx. $$ The last limit holds because of the Spectral Theorem characterization of the domain of $S$ as the set of $x$ for which $\int_{\mathbb{R}}\lambda^{2}\,d\|E(\lambda)x\|^{2} < \infty$.

In your case: Based on the phrasing of your comment it appears that you know $\mathcal{D}(A^{\star}A)$ is a core for $A$, and you want to know if $\mathcal{D}(A^{\star}A)$ is a core for $|A|$. So I'll only address the statement that $\mathcal{D}(A^{\star}A)$ is a core for $|A|$.

If $A$ is as you state, then $|A|=(A^{\star}A)^{1/2}$ has $\mathcal{D}(|A|^{2})=\mathcal{D}(A^{\star}A)$ as a core, based on the analysis for the case where $S=|A|$.