How to prove an identity (Trigonometry Angles--Pi/13)

Since $\pm 1,\pm 3,\pm 4$ are the only quadratic residues in $\mathbb{F}_{13}^*$, both sums can be related with a quadratic Gauss sum $\pmod{13}$. You just need to apply the cosine duplication formula to the first one and to consider the square of the second one.

For the first one, let $\zeta=e^{2\pi i/13}$. This is a thirteenth root of $1$, so $1+\zeta+\zeta^2+...+\zeta^{12}=0$. \begin{align} A &=\cos^2(\pi/13)+\cos^2(3\pi/13)+\cos^2(4\pi/13)\\ B &=2A-3=\cos(2\pi/13)+\cos(6\pi/13)+\cos(8\pi/13)\\ 2B &=\zeta+\zeta^{-1}+\zeta^3+\zeta^{-3}+\zeta^4+\zeta^{-4}\\ C &=-1-2B=\zeta^2+\zeta^{-2}+\zeta^5+\zeta^{-5}+\zeta^6+\zeta^{-6}\\ 2BC &=3(\zeta+\zeta^2+...+\zeta^{12})=-3\\ & 2B(2B+1) *=3 \end{align}