Block inverse of symmetric matrices
Certainly, one can use the bordering method for this (a special case of the usual formula for block inversion):
$$\begin{pmatrix}\mathbf A&\mathbf \delta\\\mathbf \delta^\top&Z\end{pmatrix}^{-1}=\begin{pmatrix}\mathbf A^{-1}+\frac{\mathbf A^{-1}\mathbf \delta\mathbf \delta^\top\mathbf A^{-1}}{\mu}&-\frac{\mathbf A^{-1}\mathbf \delta}{\mu}\\-\frac{\mathbf \delta^\top\mathbf A^{-1}}{\mu}&\frac1{\mu}\end{pmatrix}$$
where $\mathbf \delta^\top=(X\quad Y)$ and $\mu=Z-\mathbf \delta^\top\mathbf A^{-1}\mathbf \delta$.