Why $p$-adically interpolate?

This is not a question that has a short answer, because $p$-adic $L$-functions are one of the pivotal objects in modern number theory.

Let me try, though:

Riemann proved that the zeta function assumes rational values at negative odd integers. The same sort of results holds for special values of Dirichlet $L$-functions.

Let us consider Dirichlet $L$-functions $\sum_{n = 1}^{\infty} \chi(n) n^{-k}$ where $\chi$ is a Dirichlet character modulo some $p^N$. (If $\chi$ is trivial, then this gives the "$p$-deprived" zeta function, i.e. the Riemann zeta function, but with Euler factor at $p$ removed.)

Now we can think of $n \mapsto \chi(n) n^{-k}$ as a character on $(\mathbb Z/p^N)^{\times}$ with values in $\mathbb Z[\zeta]/p^N$ (where $\zeta$ is an appropriate root of unity chosen so that the values of $\chi$ land in $\mathbb Z[\zeta]$).

If $\chi,k$ and $\chi',k'$ are such that the two characters $n \mapsto \chi(n)n^{-k}$ and $n \mapsto \chi'(n) n^{\prime -k}$ are congruent mod $p^a$ (for some $a \leq N$), then looking at the Dirichlet series you might imagine that the corresponding $L$-values $L(\chi,k)$ and $L(\chi',k')$ should also be congruenct mod $p^a$ (just because the individual terms are congruent).

Of course reasoning this way with congruences isn't valid for infinite series, and if $k$ and $k'$ are in the half-plane of convergence of the $L$-functions, then the corresponding $L$-values are transcendental and congruence doesn't even make sense.

But ... at values of $k$ and $k'$ where the values are rational numbers, the congruence does hold! Of course the above "reasoning" from a term-by-term congruence of the members of the sequence is bogus, both because it is not valid for an infinite series, and the series doesn't converge at these negative values of $k$ anyway. But one can still prove the congruence. (Kummer did the case $a = 1$, i.e. just working mod $p$. Kubota and Leopoldt did the general case.)

Kummer's motivation was (among other things) that the case when $k = 0$ and $\chi$ is non-trivial has to do with the class numbers of $p$-adic cyclotomic fields (by the class number formula) and he wanted to know when these were divisible by $p$ (i.e. when primes were irregular). Using his congruence, he could instead test this by working with the trivial character but with a non-zero (negative) value of $k$. But in this case we are just looking at special values of the $\zeta$-function, which are Bernoulli numbers. This is how Kummer proved his criterion for a prime to be regular in terms of Bernoulli numbers.

So the theory of $p$-adic $L$-functions has its roots in Kummer's study of cyclotomic fields. It continues to be a central topic in the study of cyclotomic fields, and, generally speaking, $p$-adic $L$-functions form the basis for almost all modern investigations into formulas relating $L$-values and arithmetic (such as the BSD conjecture, the main conjecture of Iwasawa theory in various contexts, the Bloch--Kato conj., etc.)