How to think about the Lebesgue measure on the Gaussian Unitary Ensemble
A $n\times n$ Hermitian matrix, $M$, is determined by its entries along its diagonal and above its diagonal. Therefore only the entries $\left(M_{ij}\right)_{1\leq i \leq j \leq n}$ along and above the diagonal have to be specified. Furthermore since $M_{ii} = \overline{M}_{ii}$, i.e. the diagonal entries are equal to there complex conjugate, they are real. There are no further constraints on the entries thus the entries $\left(\left(M_{ii}\right)_{1\leq i \leq n}, \left(M_{ij}\right)_{1\leq i < j \leq n}\right)$ take values in $\mathbb{R}^n \times \mathbb{C}^{n(n-1)/2}$. Writing the real and imaginary parts of $M_{ij}$ as respectively, $M_{ij}^{(0)}$ and $M_{ij}^{(1)}$, $M$ is equivalent to a real vector $\left(\left(M_{ii}\right)_{1\leq i \leq n}, \left(\left(M_{ij}^{(0)}, M_{ij}^{(1)}\right)\right)_{1\leq i < j \leq n}\right)$ in $\mathbb{R}^n \times \left(\mathbb{R}^2\right)^{n(n-1)/2} \cong \mathbb{R}^{n^2}$.
$dM$ is the Lebesgue measure on $\mathbb{R}^2$ written in such a way as to make explicit the dependence on the components of $M$.