Prove that $\cot^2{(\pi/7)} + \cot^2{(2\pi/7)} + \cot^2{(3\pi/7)} = 5$

First of all, by noticing $\cot^2(\pi - x) = \cot^2(x)$, we can write this identity as $$\sum_{k=1,3,5} \cot^2(\frac{2\pi k}{14}) = 5$$

By writing $\cot^2(x) = \frac{1}{\sin^2(x)} - 1$, and using symmetries of $\cos$ and $\sin$ ($\cos(x)=\cos(-x)$, $\sin(\frac{\pi}{2}-x)=\cos x$), we can write this sum as follows:

$$\sum_{k=1,3,5} \frac{1}{\cos^2(\frac{\pi k}{14})} = 8$$

If we let $a_i = \cos(\frac{\pi (2i-1)}{14}), i=1,2,3$, we can write this expression as $$(*) \frac{(\sum_{i<j} a_i a_j)^2 - 2\prod a_i \sum a_i}{(\prod a_i)^2}$$

The 7'th Chebyshev Polynomial (of the first kind) vanishes exactly on $\cos(\frac{2k-1}{14}\pi)$, $1\le k \le 7$. Those roots are actually $\pm (a_1, a_2, a_3)$ and $0$, each is a simple root.

We can compute the polynomial recursively and find that it equals $$T_7(x) = 64x^7-112x^5+56x^3-7x=x(64x^6-112x^4+56x^2-7)$$ We'll work with $P_7(x)=\frac{T_7(x)}{64x}$, a monic polynomial with roots $\pm(a_1,a_2,a_3)$.

This shows, by using Vieta and the symmetry of roots (it requires some manipulation on symmetric polynomials):

1. $(\prod a_i)^2=\frac{7}{64}$ (by considering coefficient of $x^0$)
2. $(\sum_{i<j} a_i a_j)^2 - 2\prod a_i \sum a_i = \frac{56}{64}$ (by considering coefficient of $x^2$ - this one required some computation)

So the sum $(*)$ equals $\frac{56}{64} / \frac{7}{64} = 8$, which implies your identity. $\blacksquare$

EDIT: I'll describe some of the philosophy behind the answer.

The first half - I knew I wanted to you Chebyshev polynomials in some way (because its roots are related to the expression), so I did basic manipulations that helped me use the coefficients of the Chebyshev polynomial. I didn't know apriori that there are any 'good' manipulations, but I hoped and it indeed worked out.

The second half - What I really wanted is a polynomial $Q(x)$ whose roots are $a_1,a_2,a_3$. Unfortunately, I had managed only to construct the polynomial $P_7(x)$ which equals $-Q(x)Q(-x)$. Fortunately, the coefficients of $P_7$ encode enough information about the coefficients of $Q$. Explicitly, by comparing coefficients: $$P_7[X^k] = \sum_{i+j=k} (-1)^{1+j} Q[X^i]Q[X^j]$$ I used this for $k=0,2$ and it was enough. $k=0$ gave $P_7(0)=-Q(0)^2$, i.e. we have the product of the $a_i$! (up to sign, but we don't even need it.)

$k=2$ gave $P_7[X^2] = Q[X^1]^2-2Q[X^2]Q[X^0]$, which luckily was exactly the missing ingredient in calculating the rational expression $(*)$, so that's it.

EDIT 2: I feel that I need to expand on the "theory" of Chebyshev polynomial, because using it might scare people away.

The $n$'th Chebyshev polynomial of the first kind is the unique polynomial satisfying $T_n(\cos (\theta)) = \cos(n\theta)$, for any $\theta$. Evidently, $\cos(\frac{\pi}{2n}(2k+1))$ is a root for any $k$ - just plug $\theta = \frac{\pi}{2n}(2k+1))$. As $\deg T_n = n$ (see the next paragraph), there can be no other roots.

Why is $T_n$ necessarily a polynomial? Well, for $n=0$ we have $T_0 = 1$, and for $n=1$ we have $T_1(x)=x$. For $n=2$ we already need some trigonometry: $\cos(2\theta)=2\cos^2(\theta)-1$, so $T_2(x)=2x^2-1$. We can define $T_n$ recursively by trigonometric insights: $$\cos(\alpha)+\cos(\beta)=2\cos(\frac{\alpha+\beta}{2})\cos(\frac{\alpha-\beta}{2})$$ $$\implies \cos((n+1)\theta) + \cos((n-1)\theta) = 2\cos(n\theta)\cos(\theta)$$ $$\implies T_{n+1}(x) + T_{n-1}(x) = 2T_{n}(x)x$$

This is how I calculated $T_7$. In practice I just used the recurrence relation $T_{n+1}(x) = 2T_{n}(x)x-T_{n-1}$ and the table here. There are some shortcuts, since the leading coefficient of $T_n$ is $2^{n-1}$ and the last coefficient is $0$ when $n$ is odd.

Using this,

the roots of $\displaystyle z^3+z^2-3z-1=0\qquad (1)$ are $\displaystyle 2\cos\frac{2\pi}7, 2\cos\frac{4\pi}7, 2\cos\frac{6\pi}7$

If $\displaystyle\cot^2\frac{r\pi}7=u, \cos\frac{2r\pi}7=\frac{1-\tan^2\frac{r\pi}7}{1+\tan^2\frac{r\pi}7}=\frac{\cot^2\frac{r\pi}7-1}{\cot^2\frac{r\pi}7+1}=\frac{u-1}{u+1}$

$\displaystyle\implies\frac{2(u-1)}{u+1}=2\cos\frac{2r\pi}7($ where $r=1,2,3)$ will satisfy $(1)$

$\displaystyle\implies \left(\frac{2(u-1)}{u+1}\right)^3+\left(\frac{2(u-1)}{u+1}\right)^2-3\left(\frac{2(u-1)}{u+1}\right)-1=0$

On simplification, $\displaystyle 7 u^3-35 u^2+21 u-1=0$ whose roots are $\displaystyle\cot^2\frac{r\pi}7($ where $r=1,2,3)$

Now, use Vieta's formulas, to find $\displaystyle \sum \cot^2\frac{r\pi}7=\frac{35}7$

As $$\tan(2n+1)s=\frac{t^{2n+1}-\binom{2n+1}2t^{2n-1}+\cdots}{\binom{2n+1}1t^{2n}-\binom{2n+1}3t^{2n-2}+\cdots}$$ where $t=\tan s$

So, $$\tan 7s=\frac{t^7-21t^5+35t^3-7t}{7t^6-35t^4+21t-1}$$

If we put $7s=\pi,t^7-21t^5+35t^3-7t=0--->(1)$ whose roots are $\tan\frac{r\pi}7$ where $r=0,1,2,3,4,5,6$

So, the roots of $t^6-21t^4+35t^2-7=0--->(2)$ are $\tan\frac{r\pi}7$ where $r=1,2,3,4,5,6$

If we put $z=\frac1t$ (as $t\ne0,$) $\frac1{z^6}-\frac{21}{z^4}+\frac{35}{z^2}-7=0\implies z^6-5z^4+3z^2-\frac17=0$ whose roots are $\cot\frac{r\pi}7$ where $r=1,2,3,4,5,6$

So, $$z^6-5z^4+3z^2-\frac17=\prod_{1\le r\le 6}(z-\cot\frac{r\pi}7)$$

But as $\cot\frac{(7-r)\pi}7=\cot(\pi-\frac{r\pi}7)=-\cot\frac{r\pi}7$,

so $\prod_{1\le r\le 6}(z-\cot\frac{r\pi}7)$ $=\prod_{1\le r\le 3}(z-\cot\frac{r\pi}7)\prod_{4\le r\le 6}(z-\cot\frac{r\pi}7)$ $=\prod_{1\le r\le 3}(z-\cot\frac{r\pi}7)\prod_{3\ge u\ge 1}(z+\cot\frac{u\pi}7)$ (putting $7-r=u$)

$=\prod_{1\le r\le 3}(z^2-\cot^2\frac{r\pi}7)$

So,$z^6-5z^4+3z^2-\frac17$ $=z^6-z^4\sum_{1\le r\le 3}\cot^2\frac{r\pi}7$ $+z^2(\cot^2\frac{\pi}7\cot^2\frac{2\pi}7+\cot^2\frac{\pi}7\cot^2\frac{2\pi}7+\cot^2\frac{2\pi}7\cot^2\frac{3\pi}7)-\prod_{1\le r\le 3}\cot^2\frac{r\pi}7$

Comparing the coefficients of $z^4,$ we get the required identity.

Alternatively, If we put $\cot^2\frac{n\pi}7=y$ where $n=1,2,3$ or $n=7-1,7-2,7-3$ we get $y=\frac1{t^2}$ (as $\cot\frac{(7-r)\pi}7=-\cot\frac{r\pi}7$)

Replacing $t^2=\frac1y$ in $(2)$ we get $\frac1{y^3}-\frac{21}{y^2}+\frac{35}y-7=0$

or $7y^3-35y^2+21y-1=0$

So, $\sum_{1\le n\le 3}\cot^2\frac{n\pi}7=\frac{35}7=5$ using Vieta's Formulae.

The following approach is intimately related to Ofir's; you can think of this as the linear algebraic variation of his route.

Consider the perturbed Toeplitz tridiagonal matrix

$$\begin{pmatrix}2&-1&&&\\-1&2&-1&&\\&-1&\ddots&\ddots&\\&&\ddots&2&-1\\&&&-1&1\end{pmatrix}$$

which has the characteristic polynomial

$$(-1)^n\left((x-1)U_{n-1}\left(\frac{x-2}{2}\right)-U_{n-2}\left(\frac{x-2}{2}\right)\right)$$

(where $U_n(x)$ is the Chebyshev polynomial of the second kind) and the eigenvalues

$$\mu_k=4\cos^2\left(\frac{k\pi}{2n+1}\right),\qquad k=1\dots n$$

We find then that the matrix $\mathbf H=4\mathbf I-\mathbf T$ has the eigenvalues

$$\eta_k=4\sin^2\left(\frac{k\pi}{2n+1}\right),\qquad k=1\dots n$$

Now, take the case $n=3$:

$$\mathbf H=\begin{pmatrix}2&1&\\1&2&1\\&1&3\end{pmatrix}$$

We have

$$\mathbf U=\mathbf H^{-1}=\frac17\begin{pmatrix}5&-3&1\\-3&6&-2\\1&-2&3\end{pmatrix}$$

The eigenvalues of $\mathbf U$ are

$$\xi_k=\frac14\csc^2\left(\frac{k\pi}{7}\right),\qquad k=1\dots 3$$

which means the eigenvalues of $\mathbf W=4\mathbf U-\mathbf I$ are

$$\lambda_k=\cot^2\left(\frac{k\pi}{7}\right),\qquad k=1\dots 3$$

Now, with

$$\mathbf W=\frac17\begin{pmatrix}13&-12&4\\-12&17&-8\\4&-8&5\end{pmatrix}$$

you can see that the trace of $\mathbf W$ (which is also the sum of the eigenvalues of $\mathbf W$) is $5$.