Compute $\det(A^n+B^n)$

Although I don't know the real tricks behind the problem, the problem is not as terrible as it appears.

As $A$ and $B$ commute, they are simultanesouly triangularisable over $\mathbb C$. Given that $\det(B)=0$, we may assume that $A$ and $B$ are triangular matrices whose diagonal entries are respectively $a_1,a_2,a_3$ and $b_1,b_2,0$. So, the other three determinant conditions amount to \begin{align} (a_1-b_1)(a_2-b_2)a_3&=1,\tag{1}\\ (a_1+b_1)(a_2+b_2)a_3&=3,\tag{2}\\ (a_1^2+b_1^2)(a_2^2+b_2^2)a_3^2&=1.\tag{3} \end{align} Hence $a_3$ is nonzero. Now $A$ must be invertible. Otherwise, if, say, $a_2=0$, then the sum and difference of $(1)$ and $(2)$ would give $a_1b_2a_3=1$ and $b_1b_2a_3=2$, which contradict $(3)$.

Let $C=A^{-1}B$. Then $C$ is singular (because $B$ is). Let its complex eigenvalues be $x,y,0$. By the given conditions, we have \begin{align} \det(I+C)&=3\det(I-C),\\ (1+x)(1+y)&=3(1-x)(1-y),\\ 2(x+y)&= xy + 1,\tag{4} \end{align} and also \begin{align} \det(I-C)^2&=\det(I+C^2),\\ (1-x)^2(1-y)^2 &= (1+x^2)(1+y^2),\\ 4xy - 2(x+y)(xy+1) &= 0,\\ 4xy - (xy+1)^2 &= 0 \quad\text{ by } (4),\\ xy &= 1.\tag{5} \end{align} Now equations $(4)$ and $(5)$ show that $x$ and $y$ are roots of the equations $xy=1$ and $x+y=1$. As $C$ is a real matrix, nonreal eigenvalues must occur in a conjugate pair, so $\{x,y\}=\{e^{i\pi/3},e^{-i\pi/3}\}$. Substitute $b_1=e^{i\pi/3}a_1$ and $b_2=e^{-i\pi/3}a_2$ into $(1)$, we get $\det(A)=1$. Therefore $$ \det(A^n+B^n)=\det(I+C^n)\det(A)^n=|1+e^{in\pi/3}|^2=2 + 2\cos\left(\frac{n\pi}{3}\right). $$