Solution 1:

The answer is $\frac{9}{16}-\frac{1}{4\pi^{2}}$.

Let $f: x\mapsto x\cos^{2}(\pi x)$ and $\displaystyle I=\int_{0}^{3/2}f(x){\rm d}x$. The function $f$ is continuous on $[0;3/2]$ so $I$ there exist and is finite.

Note that $$\forall x\in \mathbb{R}:\quad \cos^{2}(\pi x)=\frac{1}{2}\left(\cos (2\pi x)+1\right).$$

Then, \begin{eqnarray*} \int x\cos^{2}(\pi x){\rm d}x&=&\frac{1}{2}\int x(\cos(2\pi x)+1){\rm d}x\\ &=& \frac{1}{2}\int x\cos 2\pi x {\rm d}x+\frac{1}{2}\int {\rm d}x\\ &\overset{{\rm IBP}}{=}&\frac{x\sin 2\pi x}{4\pi}-\frac{1}{4\pi}\int \sin 2\pi x{\rm d}x+\frac{x^{2}}{4}\\ &\overset{t\mapsto 2\pi x}{=}&\frac{x\sin 2\pi x}{4\pi}-\frac{1}{8\pi^{2}}\int \sin t{\rm d}t+\frac{x^{2}}{4}\\ &=&\frac{\cos t}{8\pi^{2}}+\frac{x\sin 2\pi x}{4\pi}+\frac{x^{2}}{4}+C\\ &=&\frac{2\pi x(\pi x+\sin(2\pi x))+\cos(2\pi x)}{8\pi^{2}}+C \end{eqnarray*}

Therefore, $$\color{red}{\boxed{\int_{0}^{3/2}x\cos^{2}(\pi x){\rm d}x=\frac{9}{16}-\frac{1}{4\pi^{2}}\approx 0.537}}.$$

Solution 2:

You could try to use $cos^2(\pi x) = \frac{cos(2\pi x) + 1}{2}$ before applying integration by parts.