Compute $\int_0^{\pi/2} x^2\left(\sum_{n=1}^\infty (-1)^{n-1} \cos^n(x)\cos(nx)\right)dx$
How to prove
$$I=\int_0^{\pi/2} x^2\left(\sum_{n=1}^\infty (-1)^{n-1} \cos^n(x)\cos(nx)\right)dx=\frac16\left(\frac{\pi^3}{12}-\pi\operatorname{Li}_2\left(\frac13\right)\right)$$
This problem is proposed by Cornel which can be found here where he suggested that the problem can be solved with and without harmonic series.
Here is my approach but I got stuck at the blue integral:
Using the common identity
$$ \sum_{n=1}^{\infty}p^n \cos(nx)=\frac{p(\cos(x)-p)}{1-2p\cos(x)+p^2}, \ |p|<1$$
Set $p=-\cos(x)$ we get
$$ \sum_{n=1}^{\infty}(-1)^n \cos^n(x) \cos(nx)=-\frac{2\cos^2(x)}{1+3\cos^2(x)}=-\frac23+\frac23\frac1{1+3\cos^2(x)}$$
Multiply both sides by $-x^2$ then integrate from $x=0$ to $\pi/2$ we get
$$\int_0^{\pi/2} x^2\left(\sum_{n=1}^\infty (-1)^{n-1} \cos^n(x)\cos(nx)\right)dx=\frac23\int_0^{\pi/2} x^2dx-\frac23\color{blue}{\int_0^{\pi/2}\frac{x^2}{1+3\cos^2(x)}dx}\\=\frac{\pi^3}{36}-\frac23\left(\color{blue}{\frac{\pi^3}{48}+\frac{\pi}{4}\operatorname{Li}_2\left(\frac13\right)}\right)=\frac{\pi^3}{72}-\frac{\pi}{6}\operatorname{Li}_2\left(\frac13\right)$$
I have two Questions:
1) Can we evaluate $I$ in a different way?
2) How to finish the blue integral?
My try to the blue integral is using integration by parts
$$\int\frac{dx}{1+3\cos^2(x)}=\frac12\tan^{-1}\left(\frac{\tan(x)}{2}\right)=-\frac12\tan^{-1}\left(2\cot(x)\right)$$
which gives us
$$\int_0^{\pi/2}\frac{x^2}{1+3\cos^2(x)}dx=\frac{\pi^3}{16}-\int_0^{\pi/2}x\tan^{-1}\left(\frac{\tan(x)}{2}\right)dx$$
Or
$$\int_0^{\pi/2}\frac{x^2}{1+3\cos^2(x)}dx=\int_0^{\pi/2}x\tan^{-1}\left(2\cot(x)\right)dx$$
I also tried the trick $x\to \pi/2-x$ but got complicated
Proof of the identity:
\begin{align} \sum_{n=0}^\infty p^ne^{inx}&=\sum_{n=0}^\infty\left(p e^{ix}\right)^n=\frac{1}{1-pe^{ix}},\quad |p|<1\\&=\frac{1}{1-p\cos(x)-ip\sin(x)}=\frac{1-p\cos(x)+ip\sin(x)}{1-2p\cos(x)+p^2}\\ &=\frac{1-p\cos(x)}{1-2p\cos(x)+p^2}+i\frac{p\sin(x)}{1-2p\cos(x)+p^2} \end{align}
By comparing the real and imaginary parts, we get
$$\sum_{n=\color{blue}{0}}^\infty p^n \cos(nx)=\frac{1-p\cos(x)}{1-2p\cos(x)+p^2}\Longrightarrow \sum_{n=\color{blue}{1}}^\infty p^{n-1} \cos(nx)=\frac{\cos(x)-p}{1-2p\cos(x)+p^2}$$
and
$$\sum_{n=\color{red}{0}}^\infty p^n \sin(nx)=\frac{p\sin(x)}{1-2p\cos(x)+p^2}\Longrightarrow \sum_{n=\color{red}{1}}^\infty p^n \sin(nx)=\frac{p\sin(x)}{1-2p\cos(x)+p^2}$$
We can use the following fourier series:$$\frac{1}{a+b\cos t}=\frac{1}{\sqrt{a^2-b^2}}+\frac{2}{\sqrt{a^2-b^2}}\sum_{n=1}^{\infty}\left(\frac{\sqrt{a^2-b^2}-a}{b}\right)^n\cos{(nt)},\ a>b$$ Plugging $a=5, b=3$ and $t=2x$ we get: $$\frac{1}{1+3\cos^2 x}=\frac{2}{5+3\cos(2x)}=\frac{1}{2}+\sum_{n=1}^\infty (-1)^n\left(\frac{1}{3}\right)^n\cos(2nx)$$ $$\Rightarrow \int_0^\frac{\pi}{2}\frac{x^2}{1+3\cos^2 x}dx=\frac12\int_0^\frac{\pi}{2} x^2dx+\sum_{n=1}^\infty(-1)^n \left(\frac13\right)^n\int_0^\frac{\pi}{2}x^2 \cos(2nx)dx$$ $$=\frac{\pi^3}{48}+\frac{\pi}4\sum_{n=1}^\infty \left(\frac13\right)^n\frac{1}{n^2}=\frac{\pi^3}{48}+\frac{\pi}{4}\operatorname{Li}_2\left(\frac13\right)$$
Using the series obtain above, we can also conclude that: $$\sum_{n=1}^{\infty}(-1)^n \cos^n(x) \cos(nx)=-\frac13+\frac23\sum_{n=1}^\infty \left(-\frac{1}{3}\right)^n\cos(2nx)$$
Evaluating the blue integral:
First we write
$$\frac1{1+3\cos^2(x)}=\frac{1}{5+3\cos(2x)}$$
Using the same identity in the post body
$$\sum_{n=1}^{\infty}p^n \cos(nx)=\frac{p(\cos(x)-p)}{1-2p\cos(x)+p^2}, \ |p|<1\tag1$$
But lets manipulate the denominator to have it in the form of $\frac1{5+3\cos(x)}$:
$$\frac1{1-2p\cos(x)+p^2}=\frac{-\frac{3}{2p}}{-\frac{3(1+p^2)}{2p}+3\cos(x)}$$
Now set $$-\frac{3(1+p^2)}{2p}=5\Longrightarrow p=-3,-\frac13$$
and since $|p|<1$, so we take $p=-\frac13$. Plug this value in (1) and replace $x$ by $2x$ we get
$$\frac{1}{5+3\cos(2x)}=\frac{1}{4}+\frac12\sum_{n=1}^\infty (-1)^n\left(\frac{1}{3}\right)^n\cos(2nx)\tag2$$
Multiply both sides of (2) by $x^2$ and integrate between $0$ and $\pi/2$ we get
$$\int_0^{\pi/2}\frac{x^2}{1+3\cos^2(x)}dx=\frac{\pi^3}{48}+\frac{\pi}{4}\operatorname{Li}_2\left(\frac13\right)$$
Addendum:
The identity used by @Zacky above:
$$\frac{1}{a+b\cos(x)}=\frac{1}{\sqrt{a^2-b^2}}+\frac{2}{\sqrt{a^2-b^2}}\sum_{n=1}^{\infty}\left(\frac{\sqrt{a^2-b^2}-a}{b}\right)^n\cos{(nx)},\ a>b\tag{3}$$
can be derived the same way:
$$\frac1{1-2p\cos(x)+p^2}=\frac{-\frac{b}{2p}}{-\frac{b(1+p^2)}{2p}+b\cos(x)}$$
If we set $$-\frac{b(1+p^2)}{2p}=a\tag{4}$$
we can write
$$\frac1{1-2p\cos(x)+p^2}=\frac{\frac{a}{1+p^2}}{a+b\cos x}$$
We proved above that
$$\sum_{n=1}^{\infty}p^n \cos(nx)=\frac{p(\cos(x)-p)}{1-2p\cos(x)+p^2}=-\frac12-\frac12\frac{p^2-1}{1-2p\cos(x)+p^2}$$
$$=-\frac12-\frac12 \color{red}{\frac{p^2-1}{p^2+1}}\frac{\color{red}{a}}{a+b\cos(x)}\tag5$$
From $(4)$ we find $p=\frac{\sqrt{a^2-b^2}-a}{b}$. Note that we ignored $p=\frac{\sqrt{a^2-b^2}+a}{b}$ as $|p|<1$.
Substitute this root in $(5)$ we get
$$\sum_{n=1}^\infty\left(\frac{\sqrt{a^2-b^2}-a}{b}\right)^n\cos(nx)=-\frac12-\frac12\cdot\frac{\color{red}{-\sqrt{a^2-b^2}}}{a+b\cos(x)}$$
or
$$\frac{1}{a+b\cos(x)}=\frac{1}{\sqrt{a^2-b^2}}+\frac{2}{\sqrt{a^2-b^2}}\sum_{n=1}^{\infty}\left(\frac{\sqrt{a^2-b^2}-a}{b}\right)^n\cos{(nx)}$$