Normalizer and centralizer are equivalent when $p$ is the smallest prime dividing $|G|$

You're on the right track. Since $C_G(P)$ contains $P$, and $P$ is a Sylow $p$-subgroup, it follows that the index $[N_G(P),C_G(P)]$ is prime to $p$, and divisible only by primes greater than $p$. But, the quotient group $N_G(P)/C_G(P)$ embeds in the automorphism group of $P$. (The action of $N_G(P)$ on $P$ by conjugation furnishes a homomorphism $N_G(P)\to\operatorname{Aut}P$ with kernel equal to $C_G(P)$; this is sometimes called the "N/C Lemma".) But as $P$ is cyclic, the prime divisors of $\operatorname{Aut}P$ do not exceed $P$, so $N_G(P)/C_G(P)$ must be trivial.

For the last step, recall that the automorphism group of a cyclic group $C_{p^n}$ of prime power order $p^n$ is isomorphic to $C_{p^{n-1}(p-1)}$, for odd $p$; and, for $p=2$ and $n>2$, the automorphism group is $C_2\times C_{2^{n-2}}$. (And, $\operatorname{Aut}(C_4)\cong C_2$ while $\operatorname{Aut}(C_2)$ is trivial.)