Simplify $\prod_{k=1}^5\tan\frac{k\pi}{11}$ and $\sum_{k=1}^5\tan^2\frac{k\pi}{11}$

The main building block of our solution will be the formula \begin{align*}\prod_{k=0}^{N-1}\left(x-e^{\frac{2k i\pi}{N}}\right)=x^N-1.\tag{0} \end{align*} It will be convenient to rewrite (0) for odd $N=2n+1$ in the form \begin{align*} \prod_{k=1}^{n}\left[x^2+1-2x\cos\frac{\pi k}{2n+1}\right]=\frac{x^{2n+1}-1}{x-1}. \tag{1} \end{align*} Replacing therein $x\leftrightarrow -x$ and multiplying the result by (1), we may also write \begin{align*} \prod_{k=1}^{n}\left[\left(x^2-1\right)^2+4x^2\sin^2\frac{\pi k}{2n+1}\right]=\frac{1-x^{4n+2}}{1-x^2}. \tag{2} \end{align*}

1. Setting in (1) $x=-i$, we get $$\left(2i\right)^n\prod_{k=1}^n\cos\frac{\pi k}{2n+1}=\frac{i^{2n+1}-1}{i-1} \qquad \Longrightarrow\qquad \prod_{k=1}^n2\cos\frac{\pi k}{2n+1}=1.$$

2. Setting in (2) $x=1$ and computing the corresponding limit on the right, we get $$\prod_{k=1}^n2\sin\frac{\pi k}{2n+1}=\left[\lim_{x\to 1}\frac{1-x^{4n+2}}{1-x^2}\right]^{\frac12}=\sqrt{2n+1}.$$

3. Combining the two results yields $$\boxed{\quad\prod_{k=1}^n\tan\frac{\pi k}{2n+1}=\sqrt{2n+1}\quad}$$ and to find $X$, it suffices to set $n=5$.

4. To find $Y$, let us rewrite (1) in the form (set $x=-e^{i\gamma}$) $$\prod_{k=1}^n \left[2\cos\gamma+2\cos\frac{\pi k}{2n+1}\right]=\frac{\cos\frac{\left(2n+1\right)\gamma}{2}}{\cos\frac{\gamma}{2}}$$ Taking the logarithm and differentiating twice with respect to $\gamma$, we find $$\sum_{k=1}^{n}\frac{1}{\left(\cos\gamma+\cos\frac{\pi k}{2n+1}\right)^2} =-\frac{1}{\sin\gamma}\frac{\partial}{\partial \gamma}\left(\frac{1}{\sin\gamma}\frac{\partial}{\partial \gamma}\ln \frac{\cos\frac{\left(2n+1\right)\gamma}{2}}{\cos\frac{\gamma}{2}}\right).\tag{3}$$

5. Computing the right side of (3) and setting therein $\gamma=\frac{\pi}{2}$, we finally arrive at $$\sum_{k=1}^{n}\frac{1}{\cos^2\frac{\pi k}{2n+1}}=2n(n+1)\qquad \Longrightarrow\quad \boxed{\quad\sum_{k=1}^{n}\tan^2\frac{\pi k}{2n+1}=n(2n+1)\qquad}$$ This yields $Y=55$.

Like Roots of a polynomial whose coefficients are ratios of binomial coefficients,

$$\tan(2n+1)x=\dfrac{\binom{2n+1}1\tan x-\binom{2n+1}3\tan^3x+\cdots}{1-\binom{2n+1}2\tan^2x+\cdots}$$

If $\tan(2n+1)x=0,(2n+1)x=r\pi$ where $r$ is ay integer

$\implies x=\dfrac{r\pi}{2n+1}$ where $r\equiv-n,-(n-1),\cdots,0,1,\cdots,n\pmod{2n+1}$

So, the roots of $\displaystyle\binom{2n+1}1\tan x-\binom{2n+1}3\tan^3x+\cdots+(-1)^{n-1}\binom{2n+1}{2n-1}\tan^{2n-1}x+(-1)^n\tan^{2n+1}x=0$ $\displaystyle\iff\tan^{2n+1}x-\binom{2n+1}2\tan^{2n-1}x+\cdots+(-1)^n(2n+1)\tan x=0$

are $\tan x$ where $x=\dfrac{r\pi}{2n+1}$ where $r\equiv-n,-(n-1),\cdots,0,1,\cdots,n\pmod{2n+1}$

So, the roots of $\displaystyle\tan^{2n}x-\binom{2n+1}2\tan^{2n-2}x+\cdots+(-1)^n(2n+1)=0$

are $\tan x$ where $x=\dfrac{r\pi}{2n+1}$ where $r\equiv\pm1,\pm2,\cdots,\pm n\pmod{2n+1}$

As $\tan(-A)=-\tan A,$

the roots of $\displaystyle t^nx-\binom{2n+1}2t^{n-1}x+\cdots+(-1)^n(2n+1)=0\ \ \ \ (1)$

are $\tan^2x$ where $x=\dfrac{r\pi}{2n+1}$ where $r\equiv1,2,\cdots,n\pmod{2n+1}$

Using Vieta's formula on $(1),$ $$\sum_{r=1}^n\tan^2\dfrac{r\pi}{2n+1}=\dfrac{\binom{2n+1}2}1=\cdots$$

and $$(-1)^n\prod_{r=1}^n\tan^2\dfrac{r\pi}{2n+1}=(-1)^n(2n+1)\iff\prod_{r=1}^n\tan^2\dfrac{r\pi}{2n+1}=(2n+1)$$

Now, $\tan\dfrac{r\pi}{2n+1}>0$ for $\dfrac\pi2>\dfrac{r\pi}{2n+1}>0\iff2n+1>2r>0$

$$\implies\prod_{r=1}^n\tan\dfrac{r\pi}{2n+1}=\sqrt{2n+1}$$