Prove that $\lim_{N\rightarrow\infty}(1/N)\sum_{n=1}^N f(nx)=\int_{0}^1f(t)dt$

Using the hint you were given, it is easy to verify that $$ \int_0^1 e^{2\pi i k t} \, \mathrm{d}t = \left\{ \begin{array}{lr} 1 & :\ k = 0 \\ 0 & :\ k \neq 0 \end{array} \right. $$ Similarly, for $k \neq 0$ and irrational $x [0,1]$, using geometric series $$ \begin{align*} \lim_{N \to \infty} \frac{1}{N} \left| \sum_{n=1}^N e^{2 \pi i k n x} \right| &= \lim_{N \to \infty} \frac{1}{N} \left| e^{2 \pi i k x} \frac{e^{2 \pi i k N x} - 1}{e^{2 \pi i k x} - 1} \right| \\ &\leq \lim_{N \to \infty} \frac{1}{N} \frac{2}{|e^{2 \pi i k x} - 1|} \\ &= 0 \end{align*} $$ noting that since $x$ is irrational $e^{2 \pi i k x} \neq 1$ for any $k \neq 0$. On the other hand, for $k = 0$ we have $e^0 = 1$, so $$ \lim_{N \to \infty} \frac{1}{N} \sum_{n = 1}^N 1 = 1. $$ It follows that $$ \lim_{N \to \infty} \frac{1}{N} \sum_{n = 1}^N e^{2 \pi i k n x} = \int_0^1 e^{2 \pi i k t} \, \mathrm{d}t. $$

Now for any continuous function $f$ on $\mathbb{R}$ with period $1$, there is a sequence of complex numbers $\{c_k\}_{-\infty}^\infty$ such that $$ f(t) = \sum_{k = -\infty}^\infty c_k e^{2 \pi i k t}. $$ So with a little justification of the interchange between sum and integral, $$ \begin{align*} \int_0^1 f(t) \, \mathrm{d}t &= \int_0^1 \sum_{k = -\infty}^\infty c_k e^{2 \pi i k t} \, \mathrm{d}t \\ &= \sum_{k = -\infty}^\infty c_k \int_0^1 e^{2 \pi i k t} = c_0. \end{align*} $$ And correspondingly, for irrational $x \in [0,1]$, $$ \begin{align*} \lim_{N \to \infty} \frac{1}{N} \sum_{n = 1}^N f(n x) &= \lim_{N \to \infty} \frac{1}{N} \sum_{n = 1}^N \sum_{k = -\infty}^\infty c_k e^{2 \pi i k n x} \\ &= \sum_{k = -\infty}^\infty c_k \lim_{N \to \infty} \frac{1}{N} \sum_{n = 1}^N e^{2 \pi i k n x} \\ &= c_0. \end{align*} $$

It follows that $$ \lim_{N \to \infty} \frac{1}{N} \sum_{n = 1}^N f(n x) = \int_0^1 f(t)\, \mathrm{d}t. $$