How to calculate the gradient of log det matrix inverse?

I assume that you are asking for the derivative with respect to the elements of the matrix. In this cases first notice that

$$\log \det X^{-1} = \log (\det X)^{-1} = -\log \det X$$

and thus

$$\frac{\partial}{\partial X_{ij}} \log \det X^{-1} = -\frac{\partial}{\partial X_{ij}} \log \det X = - \frac{1}{\det X} \frac{\partial \det X}{\partial X_{ij}} = - \frac{1}{\det X} \mathrm{adj}(X)_{ji} = - (X^{-1})_{ji}$$

since $\mathrm{adj}(X) = \det(X) X^{-1}$ for invertible matrices (where $\mathrm{adj}(X)$ is the adjugate of $X$, see http://en.wikipedia.org/wiki/Adjugate).

Or you can check section A.4.1 of the book Stephen Boyd, Lieven Vandenberghe, Convex Optimization for an alternative solution, where they compute the gradient without using the adjugate.