Find the inverse of a submatrix of a given matrix

Note that $A$ differs from $\begin{bmatrix} 1 & 0 \\ 0 & B \end{bmatrix}$ by at most a rank two "update". Therefore formulas which "update" the matrix inverse $A^{-1}$ can be used to give an inverse of $B$:

$$ \begin{bmatrix} 1 & 0 \\ 0 & B \end{bmatrix}^{-1} = \begin{bmatrix} 1 & 0 \\ 0 & B^{-1} \end{bmatrix} $$

A rank one update to $A$ corresponds to a rank one update to $A^{-1}$ by the Sherman-Morrison formula:

$$ (A + uv^T)^{-1} = A^{-1} - \frac{A^{-1}uv^T A^{-1} }{1 + v^T A^{-1} u} $$

The more general Woodbury formula could be used here to give the inverse of the rank two update to $A$ directly as a rank two update to $A^{-1}$.

$$ \left(A+UCV \right)^{-1} = A^{-1} - A^{-1}U \left(C^{-1}+VA^{-1}U \right)^{-1} VA^{-1} $$

The parameter $C$ above is redundant in the sense that we can take it to be the identity (and this is often done) choosing $U,V$ to "absorb" any factor $C$. In the case at hand:

$$ A = \begin{bmatrix} a & v_1 & v_2 & v_3 \\ u_1 & & & \\ u_2 & & B & \\ u_3 & & & \end{bmatrix} $$

we can take $U = \begin{bmatrix} 1 & 0 \\ 0 & u_1 \\ 0 & u_2 \\ 0 & u_3 \end{bmatrix}$ and $V = \begin{bmatrix} 0 & v_1 & v_2 & v_3 \\ 1&0&0&0 \end{bmatrix}$, so that:

$$ UV = \begin{bmatrix} 0 & v_1 & v_2 & v_3 \\ u_1 & 0 & 0 & 0 \\ u_2 & 0 & 0 & 0 \\ u_3 & 0 & 0 & 0 \end{bmatrix} $$

Then $A - UV = \begin{bmatrix} a & 0 \\ 0 & B \end{bmatrix}$ and provided $a \neq 0$ the update can be done by inverting $(I - VA^{-1}U)$, a $2\times 2$ matrix.

For the small sizes of matrices $A$ and $B$ (resp. $4\times 4$ and $3\times 3$) such computations may promise little improvement over a direct computation of $B^{-1}$. As the Comments indicated interest in removing multiple $k$ columns and rows from larger $n\times n$ matrices, the Woodbury formula can be used for this. However one needs to keep in mind the cost of inverting the $2k\times 2k$ matrix $I + VA^{-1}U$ when $2k$ is a significant fraction of original size $n$.

Maybe you can use one of the wikipedia pages:

http://en.wikipedia.org/wiki/Invertible_matrix#Blockwise_inversion

or

http://en.wikipedia.org/wiki/Block_matrix_pseudoinverse

The pseudoinverse will have to be used in case B is not invertible. For instance the case in the other answer by 5xum.