Proving that $\cos\frac{2\pi}{13}+\cos\frac{6\pi}{13}+\cos\frac{8\pi}{13}=\frac{\sqrt{13}-1}{4}$
Solution 1:
Let $t = \frac{\pi}{13}$
then you know you want $$\cos(2t)+\cos(6t)+\cos(8t)$$
and if you call that $x$, square it, then you should have $$x^2 = \cos^2(2t)+\cos^2(6t)+\cos^2(8t)[\cos(2t)+\cos(4t)+\cos(6t)+\cos(8t)+\cos(10t)+\cos(12t)]$$
Note that,
$$[\cos(2t)+\cos(4t)+\cos(6t)+\cos(8t)+\cos(10t)+\cos(12t)] = -0.5$$
Hence,
$$x^2 = \cos^2(2t)+\cos^2(6t)+\cos^2(8t) - 0.5 $$
By the law of cosines,
$$x^2 = \frac{1}{2}(\cos(4t)+\cos(12t)+\cos(16t) + 3) - 0.5 $$
Hence, $$2x^2 = \cos(4t)+\cos(12t)+\cos(16t) + 2 $$
and as we know $$x = \cos(2t) + \cos(6t) + \cos(8t)$$
Add $$2x^2 + x = \frac{-1}{2} +2 = \frac{3}{2}$$
Then we have,
$$4x^2 + 2x -3 = 0$$
Use Quadratic formula, then you will get two solutions, but you know the answer is positive.
Solution 2:
Let $\cos\frac{2\pi}{13}+\cos\frac{6\pi}{13}+\cos\frac{8\pi}{13}=x$ and $\cos\frac{4\pi}{13}+\cos\frac{10\pi}{13}+\cos\frac{12\pi}{13}=y$.
Hence, $x=2\cos\frac{5\pi}{13}\cos\frac{3\pi}{13}+\cos\frac{6\pi}{13}>0$.
Now, $$x+y=\cos\frac{2\pi}{13}+\cos\frac{4\pi}{13}+\cos\frac{6\pi}{13}+\cos\frac{8\pi}{13}+\cos\frac{10\pi}{13}+\cos\frac{12\pi}{13}=$$ $$=\tfrac{2\sin\frac{\pi}{13}\cos\frac{2\pi}{13}+2\sin\frac{\pi}{13}\cos\frac{4\pi}{13}+2\sin\frac{\pi}{13}\cos\frac{6\pi}{13}+2\sin\frac{\pi}{13}\cos\frac{8\pi}{13}+2\sin\frac{\pi}{13}\cos\frac{10\pi}{13}+2\sin\frac{\pi}{13}\cos\frac{12\pi}{13}}{2\sin\frac{\pi}{13}}=$$ $$=\tfrac{\sin\frac{3\pi}{13}-\sin\frac{\pi}{13}+\sin\frac{5\pi}{13}-\sin\frac{3\pi}{13}+\sin\frac{7\pi}{13}-\sin\frac{5\pi}{13}+\sin\frac{9\pi}{13}-\sin\frac{7\pi}{13}+\sin\frac{11\pi}{13}-\sin\frac{9\pi}{13}+\sin\frac{13\pi}{13}-\sin\frac{11\pi}{13}}{\sin\frac{\pi}{13}}=-\frac{1}{2}$$ and $$xy=\left(\cos\frac{2\pi}{13}+\cos\frac{6\pi}{13}+\cos\frac{8\pi}{13}\right)\left(\cos\frac{4\pi}{13}+\cos\frac{10\pi}{13}+\cos\frac{12\pi}{13}\right)=$$ $$=\frac{3}{2}\left(\cos\frac{2\pi}{13}+\cos\frac{4\pi}{13}+\cos\frac{6\pi}{13}+\cos\frac{8\pi}{13}+\cos\frac{10\pi}{13}+\cos\frac{12\pi}{13}\right)=-\frac{3}{4}.$$ Id est, $x$ and $y$ are roots of the following equation $$z^2+\frac{1}{2}z-\frac{3}{4}=0$$ and since $x>0$, we obtain $x=\frac{\sqrt{13}-1}{4}$.
Done!