How to divide by a matrix

Note that applying function to a matrix is meant in sense of series, that is if $$ \phi(z) = c_0 + c_1 z + c_2 z^2 + \dots $$ then $$ \phi(\mathbf A) = c_0 \mathbf I + c_1 \mathbf A + c_2 \mathbf A^2 + \dots. $$ Observe that $$ \phi(z) = \frac{e^z - 1}{z} = 1 + \frac{z}{2} + \frac{z^2}{6} + \dots = \sum_{k = 1}^\infty \frac{z^{k-1}}{k!}. $$ Also, to your question, dividing by a matrix is multiplying with its inverse, but that depends on commutativity, there's basically two ways to divide on a matrix - on the right and on the left. Luckily, a matrix commute with any power of itself, so there's no difference on which side do you write the $\mathbf A^{-1}$: $$ \phi(\mathbf A) = \mathbf A^{-1} (e^\mathbf{A} - \mathbf I). $$

Note that since $$ \exp(A)=I+A+\frac1{2!}A^2+\frac1{3!}A^3+\cdots $$ then $A$ and $\exp(A)$ commute.

Thus also $\exp(A)-I$ and $A^{-1}$ commute.

Try writing out the series for $exp$ and simplify the equation before you evaluate!

You wouldn't have to worry about dividing!

If you look at the series expansion of this function at $z=0$, you find there are only positive powers of $z$.

In any case, naively $\frac{1}{z}=z^{-1}$ just translates to the inverse, which exists for many matrices.

For all these translations, note, however, that $xy=yx$ need not hold for matrices. But if a function only involves one matrix variable ($z$ here), there is no problem.