Is there a non-matrix Lie group?

There are no non-matrix Lie groups whose dimension is $1$ or $2$. On the other hand, consider the quotient of the Heisenberg group$$\left\{\begin{pmatrix}1&a&c\\0&1&b\\0&0&1\end{pmatrix}\,\middle|\,a,b,c\in\mathbb R\right\}$$by the normal subgroup$$\left\{\begin{pmatrix}1&0&m\\0&1&0\\0&0&1\end{pmatrix}\,\middle|\,m\in\mathbb Z\right\}.$$It is a non-matrix three-dimensional Lie group.

There is the metaplactic group, which is the unique connected double cover of the symplectic group.

Lie groups are smooth manifolds. They may or may not have matrix representations. For example, the universal cover of $\mathbf{SL}_2(\mathbf{R})$ is a Lie group that is not a matrix Lie group.