Are there p-adic manifolds?

Yes, there are. One source to learn about them is the second half of Serre's book Lie Algebras and Lie Groups, and another is Peter Schneider's recent book $p$-adic Lie Groups, which is very nice.

They are called locally $p$-adic analytic manifolds, or more generally, if $k$ is any non-Archimedean field, locally $k$-analytic manifolds. The definition is formally the same as a real or complex analytic manifold, but one uses the word "local" because, for non-Archimedean fields, functions given locally by power series do not need to be given globally by a power series. This is in contrast to rigid analytic spaces, which are more akin to schemes, and have as their coordinate rings certain "affinoid" algebras of convergent power series. Much of the theory for complex manifolds carries over to locally analytic manifolds (as is made clear in Serre's book), although of course there are some differences. For example, paracompact locally analytic manifolds are "strictly paracompact," meaning that every open cover admits a refinement by pairwise disjoint opens. But one has versions of the inverse and implicit function theorems, tangent bundles, Lie algebras for locally $k$-analytic groups, etc.. The notes mentioned in the comments are on various theories of non-Archimedean analytic spaces, which are somewhat more analogous to complex analytic spaces in general.

If $k$ is a non-Archimedean field and $X$ is a smooth, separated $k$-scheme of finite type, then $X(k)$, the set of $k$-rational points, has a canonical locally $k$-analytic structure, and $X\rightsquigarrow X(k)$ is functorial in $X$. This is the same as for schemes over $\mathbf{C}$. I'm not sure what other connections with algebraic geometry you're referring to.