Second Derivative of a Determinant

To keep things simple, assume $A=A(t)$ is a function of a parameter $t$, and we are after the derivatives at zero. Also assume that $A(t)$ is smooth enough in $t$ and that $A(0)$ is invertible. While we are at it, let's replace $A(t)$ by $B(t)=A(0)^{-1}A(t)$ so that $B(0)=I$.

Then $$B(t)=I+tB'(0)+\frac{t^2}2B''(0)+\cdots$$ for some matrices $B_1$ and $B_2$. We now expand $\det B(t)$ by the formula in terms of permutations and their signs. A permutation moving three or more points will lead to a term with a factor of $t^3$ which we can neglect. The identity permutation yields the product of the diagonal elements of $B(t)$ which is \begin{align} &1+t\sum_{i=1}^n B'(0)_{ii} +\frac{t^2}2\sum_{i=1}^n B''(0)_{ii} +t^2\sum_{1\le i<j\le n}B'(0)_{ii}B'(0)_{jj}+\cdots\\ &=1+t\text{ tr}\,B'(0)+\frac{t^2}2\text{ tr}\,B''(0) +t^2\sum_{1\le i<j\le n}B'(0)_{ii}B'(0)_{jj}+\cdots \end{align} The only other permutations counting are the involutions, which together contribute $$-t^2\sum_{1\le i<j\le n}B'(0)_{ij}B'(0)_{ji}+\cdots.$$

The second derivative of $\det B(t)$ at zero is therefore $$\text{tr}\,B''(0)+2\sum_{1\le i<j\le n} (B'(0)_{ii}B'(0)_{jj}-B'(0)_{ij}B'(0)_{ji}).$$ The sum here is the "second trace" of $B'(0)$, the coefficient of $X^{n-2}$ in its characteristic polynomial.

Now if we like we can write $A(t)=A(0)B(t)$ and get a formula for the second derivative of $\det A(t)$.

I think your derivation is good up until you take the derivative of $A^{-1}$. From formula (40) of the Matrix Cookbook the derivative is:

$$\partial \left(X^{-1}\right) = - X^{-1} \left( \partial X \right) X^{-1}$$

so instead of a second term of $\operatorname{tr} \left( A_\alpha^{-1} A_\alpha \right)$ it should instead be $-\operatorname{tr} \left( A^{-1} A_\alpha A^{-1} A_\alpha \right)$. The final result then is:

$$\det \left(A\right) \left( \left(\operatorname{tr}\left(A^{-1} A_\alpha\right)\right)^2 + \operatorname{tr}\left(A^{-1} A_{\alpha\alpha}\right)-\operatorname{tr}\left(A^{-1}A_\alpha A^{-1} A_\alpha\right)\right)$$

It is a simple matter to confirm this formula symbolically for small examples in your CAS of choice (for a Mathematica implementation see my answer to a similar question on MSE).

As pointed out by Carl, your mistake is to permute the inverse and derivative operator:

$$\partial_\alpha(A^{-1}) \neq (\partial_\alpha A)^{-1} \, .$$

You can easily generalize it for whatever two differentiating arguments ($\alpha$ and $\beta$). Starting from Jacobi's formula:

$$ \partial_{\alpha}(\mathrm{det}(A)) = \mathrm{det}(A) \, \mathrm{tr}\left(A^{-1} \, \partial_\alpha A\right) \, $$

applying on it the chain rule:

$$ \partial_{\alpha\beta}(\mathrm{det}(A)) = \partial_\beta(\mathrm{det}(A)) \, \mathrm{tr}\left(A^{-1} \, \partial_\alpha A\right) + \mathrm{det}(A) \, \mathrm{tr}\left(\partial_\beta \left(A^{-1} \, \partial_\alpha A\right)\right) \, ,$$

using Jacobi's formula on the first $\beta$ derivative and applying the chain rule on the second:

$$ \partial_{\alpha\beta}(\mathrm{det}(A)) = \mathrm{det}(A) \, \mathrm{tr}\left(A^{-1} \, \partial_\beta A\right)\,\mathrm{tr}\left(A^{-1} \, \partial_\alpha A\right) + \mathrm{det}(A) \, \mathrm{tr}\left(\partial_\beta \left(A^{-1}\right) \, \partial_\alpha A\right) + \mathrm{det}(A) \, \mathrm{tr}\left(A^{-1} \, \partial_{\alpha\beta} A\right)\, ,$$

and finally, factoring out the determinant and applying the relation for the inverse $\partial_\beta(A^{-1}) = -A^{-1} \, \partial_\beta A \, A^{-1}$:

$$ \partial_{\alpha\beta}(\mathrm{det}(A)) = \mathrm{det}(A) \, \left[\mathrm{tr}\left(A^{-1} \, \partial_\beta A\right)\,\mathrm{tr}\left(A^{-1} \, \partial_\alpha A\right) - \mathrm{tr}\left(A^{-1} \, \partial_\beta A \, A^{-1} \, \partial_\alpha A\right) + \mathrm{tr}\left(A^{-1} \, \partial_{\alpha\beta} A\right)\right] \, , $$

which is the same as what Carl gave for $\beta=\alpha$:

$$ \partial_{\alpha\alpha}(\mathrm{det}(A)) = \mathrm{det}(A) \, \left[\left(\mathrm{tr}\left(A^{-1} \, \partial_\alpha A\right)\right)^2 - \mathrm{tr}\left(A^{-1} \, \partial_\alpha A \, A^{-1} \, \partial_\alpha A\right) + \mathrm{tr}\left(A^{-1} \, \partial_{\alpha\alpha} A\right)\right] \, . $$