Solution to a matrix-valued ODE is invertible at all times assuming it is at a given time.

It is a well-known in Linear ODE theory equation for fundamental matrix of linear ODE system (please see https://en.wikipedia.org/wiki/Fundamental_matrix_(linear_differential_equation)) The prove of its non-singularity is also can be found in any ODE book.

The schematic proof:

Determinant of the fundamental matrix is a Wronski determinant $W(t) = det(Z(t))$. One can show that it satisfies simple one dimentional linear ODE

$W'(t) = tr(A(t))\cdot W(t)$

which can be solved directly: $W(t) = W(t_0)\cdot \exp\left(\int_{t_0}^t tr(A(s))ds\right)$. Therefore, $W(t)$ is not equals to zero at any $t$ as long it takes place at $t=t_0$.