Calculation of $\int^{\pi/2}_{0}\cos^{n}(x)\cos (nx)\,dx$, where $n\in \mathbb{N}$

Your start is not bad, but you can get it simpler by using the evenness of $\cos$,

$$\int_0^{\pi/2} \cos^n x \cos (nx)\,dx = \frac12 \int_{-\pi/2}^{\pi/2} \cos^n x\cos (nx)\,dx.$$

Since the sine is an odd function, we can replace $\cos (nx)$ with $e^{inx}$:

$$\begin{align} \int_0^{\pi/2} \cos^n x \cos (nx)\,dx &= \frac12\int_{-\pi/2}^{\pi/2}\cos^n x \, e^{inx}\,dx\\ &= \frac{1}{2^{n+1}} \int_{-\pi/2}^{\pi/2} \left(e^{ix}+e^{-ix}\right)^n e^{inx}\,dx\\ &= \frac{1}{2^{n+1}} \int_{-\pi/2}^{\pi/2} \left(e^{2ix}+1\right)^n\,dx. \end{align}$$

Next we substitute $\varphi = 2x$ and get

$$ \int_0^{\pi/2} \cos^n x \cos (nx)\,dx = \frac{1}{2^{n+2}} \int_{-\pi}^\pi \left(e^{i\varphi}+1\right)^n\,d\varphi. $$

Now setting $z = e^{i\varphi}$ gets us a nice contour integral over the unit circle,

$$\begin{align} \int_0^{\pi/2} \cos^n x \cos (nx)\,dx &= \frac{1}{2^{n+2}} \int_{\lvert z\rvert = 1} (z+1)^n\,\frac{dz}{iz}\\ &= \frac{\pi}{2^{n+1}} \end{align}$$

by the Cauchy integral formula.

Hint:

Put $$I_n=\int_{0}^{\frac{\pi}{2}}\cos^nx\cos (nx)dx=\int_{0}^{\frac{\pi}{2}}\cos^{n-1}x\cos[(n-1)x]dx-\int_{0}^{\frac{\pi}{2}}\cos^{n-1}x\sin(nx)\sin xdx=I_{n-1}+\int_{0}^{\frac{\pi}{2}}\sin(nx)d(\frac{\cos^nx}{n})=I_{n-1}-I_{n} $$

$$\to I_n=\frac{1}{2}I_{n-1}=\cdots=\frac{1}{2^{n}}I_0=\frac{\pi}{2^{n+1}}$$

$\newcommand{\angles}[1]{\left\langle\, #1 \,\right\rangle} \newcommand{\braces}[1]{\left\lbrace\, #1 \,\right\rbrace} \newcommand{\bracks}[1]{\left\lbrack\, #1 \,\right\rbrack} \newcommand{\ceil}[1]{\,\left\lceil\, #1 \,\right\rceil\,} \newcommand{\dd}{{\rm d}} \newcommand{\ds}[1]{\displaystyle{#1}} \newcommand{\expo}[1]{\,{\rm e}^{#1}\,} \newcommand{\fermi}{\,{\rm f}} \newcommand{\floor}[1]{\,\left\lfloor #1 \right\rfloor\,} \newcommand{\half}{{1 \over 2}} \newcommand{\ic}{{\rm i}} \newcommand{\iff}{\Longleftrightarrow} \newcommand{\imp}{\Longrightarrow} \newcommand{\pars}[1]{\left(\, #1 \,\right)} \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}} \newcommand{\pp}{{\cal P}} \newcommand{\root}[2][]{\,\sqrt[#1]{\vphantom{\large A}\,#2\,}\,} \newcommand{\sech}{\,{\rm sech}} \newcommand{\sgn}{\,{\rm sgn}} \newcommand{\totald}[3][]{\frac{{\rm d}^{#1} #2}{{\rm d} #3^{#1}}} \newcommand{\verts}[1]{\left\vert\, #1 \,\right\vert}$ $\ds{\int_{0}^{\pi/2}\cos^{n}\pars{x}\cos\pars{nx}\,\dd x:\ {\large ?}.\quad n \in {\mathbb N}}$.

With $\ds{0 < \epsilon < 1}$ $\ds{\pars{~\mbox{we'll take at the end the limit}\ \epsilon \to 0^{+}~}}$: \begin{align}&\color{#66f}{\large\int_{0}^{\pi/2}\cos^{n}\pars{x}\cos\pars{nx} \,\dd x}=\Re \int_{\verts{z}\ =\ 1\atop{\vphantom{\Huge A}0\ <\ {\rm Arg}\pars{z}\ <\ \pi/2}} \pars{z^{2} + 1 \over 2z}^{n}z^{n}\,{\dd z \over \ic z} \\[3mm]&={1 \over 2^{n}}\,\Im \int_{\verts{z}\ =\ 1\atop{\vphantom{\Huge A}0\ <\ {\rm Arg}\pars{z}\ <\ \pi/2}} {\pars{z^{2} + 1}^{n} \over z}\,\dd z =-\,{1 \over 2^{n}}\,\Im\int_{1}^{\epsilon} {\pars{-y^{2} + 1}^{n} \over \ic y}\,\ic\,\dd y \\[3mm]&\phantom{=}\left.\mbox{}-{1 \over 2^{n}}\,\Im\int_{\pi/2}^{0} {\pars{z^{2} + 1}^{n} \over z}\,\dd z\, \right\vert_{\,z\ =\ \epsilon\expo{\ic\theta}} -{1 \over 2^{n}}\,\Im\int_{\epsilon}^{1} {\pars{x^{2} + 1}^{n} \over x}\,\dd x \\[3mm]&=-\,{1 \over 2^{n}}\Im\int_{\pi/2}^{0}\ic\,\dd\theta =\color{#66f}{\Large{\pi \over 2^{n + 1}}} \end{align}