Your sequence is $$S_i = (2+\sqrt3)^{3^{i+1}}+(2-\sqrt3)^{3^{i+1}}$$ Proof: check that $S_0=52$ and $S_{i+1}=S_i^3-3S_i$.

So $S_{p-1}=S_0\bmod N$ iff $$(2+\sqrt3)^{3^p} +(2-\sqrt3)^{3^p}=(2+\sqrt3)^3+(2-\sqrt3)^3\bmod N$$

If $N$ is prime then

  • $N=1\bmod 3$ so $1+\sqrt{-3}^N= (1+\sqrt{-3})^N=(2\zeta_3)^N=2 \zeta_3 =1+\sqrt{-3}\bmod N$

  • $N=1\bmod 4$ so $\sqrt{3}^N = (-i\sqrt{-3})^N= (-i)^N \sqrt{-3}^N= -i \sqrt{-3}=\sqrt3\bmod N$

  • $(2+\sqrt3)^N = 2^N+(\sqrt3)^N=2+\sqrt3\bmod N$

    Similarly $(2-\sqrt3)^N=2-\sqrt3\bmod N$

  • And hence $$S_{p-1}=(2+\sqrt3)^{2N+1}+(2-\sqrt3)^{2N+1}=(2+\sqrt3)^3+ (2-\sqrt3)^3=S_0\bmod N$$

That's a good start. But for the other direction, I don't see why $S_{p-1}=S_0\bmod N$ would imply that $N$ is not composite.