Is $\int |x\rangle \langle x|dx$ Mathematical?
Solution 1:
Yes, this does make sense in the context of "rigged Hilbert spaces", e.g., something like what is occasionally called a Gelfand triple $H^{+1}\subset L^2\subset H^{-1}$ of Sobolev spaces on an interval in $\mathbb R$. Somehow Dirac had a wonderful intuition in this direction already prior to 1930. Also, the possibility of writing "integral kernels" for all mappings was eventually systematized into L. Schwartz' Kernel Theorem, and A. Grothendieck's nuclear spaces. Perhaps the most direct way to make things completely precise is as follows.
Let $L^2=L^2[a,b]$ be the usual space of square-integrable functions, which we know is also the completion of the space of test functions on $[a,b]$ with respect to the $L^2$ norm. Let $H^1=H^1[a,b]$ be the completion of test functions with respect to the (Sobolev) norm $|f|^2_{H^1}=\langle f-f'',f\rangle$. The injection $j:H^{-1}\to L^2$ has an adjoint $j^*$, and we identify $L^2$ with its own dual (but not the others!), obtaining $j^*:L^2\to H^{-1}$ where $H^{-1}=(H^1)^*$.
Dirac delta at a point $x_o$ in $[a,b]$ is provably in $H^{-1}$.
As a small part of some version of Schwartz' Kernel Theorem in this setting, with some vector-valued integral justification, the computation you quote is exactly the verification that the kernel for the identity map is "Dirac delta on the diagonal" in $[a,b]\times[a,b]$.
(The bra-ket notation can be rewritten in terms of tensors and tensor products if one desires, making it look less physics-y.)
Solution 2:
Here is my understanding. But I know very little about QM, so this explanation may be incorrect.
Consider the multiplication operator $A$ defined on $L^2[-a, a]$ by
$$A\varphi(x) = x\varphi(x). $$
(In quantum mechanics, $A$ corresponds to the position operator.) Since this operator is bounded with the spectrum $\sigma(A) = [-a, a]$, by the spectral theorem there corresponds a projection-valued measure $E$ on $\sigma(A)$ such that
$$ A = \int_{[-a, a]} \lambda \, dE_{\lambda}. $$
This measure admits the resolution of the identity:
$$ 1 = \int_{[-a, a]} dE_{\lambda}. $$
Now heuristically, Dirac delta $\delta_{\lambda}$ is an eigenfunction of $A$ corresponding to the eigenvalue $\lambda$. Since we know that $E_{\lambda} = \mathbf{1}_{(-\infty, \lambda]}(A)$, we can think that $dE_{\lambda} \approx \mathbf{1}_{\{\lambda\}}(A)$ is a projection to the eigenspace corresponding to $\lambda$, which should be spanned by $\delta_{\lambda}$. Thus it is not unreasonable to carelessly write
$$ dE_{\lambda} \approx | \delta_{\lambda} \rangle\langle \delta_{\lambda} | \, d\lambda, $$
where $d\lambda$ comes from the normalization.