Must a ring homomorphism between $\mathbb Z_p$-algebras be a $\mathbb Z_p$-algebra homomorphism?

No, this is not true, even in the case $A=\mathbb{Z}_p$. In that case, you are asking whether any ring $B$ has at most one homomorphism from $\mathbb{Z}_p$ (since if there were two you could use one to make $B$ a $\mathbb{Z}_p$-algebra and the other as your $f$). But for instance, take $B=\mathbb{C}$. There are lots of homomorphisms $\mathbb{Z}_p\to\mathbb{C}$, because there are lots of different embeddings of $\mathbb{Q}_p$ into $\mathbb{C}$. (For instance, if you pick a transcendance basis for $\mathbb{Q}_p$ over $\mathbb{Q}$, you can map it to any family of algebraically independent elements of $\mathbb{C}$, and then can extend to an embedding of all of $\mathbb{Q}_p$ since $\mathbb{C}$ is algebraically closed.)

Here is another way to see it. Consider the two natural inclusion maps $\mathbb{Z}_p\to\mathbb{Q}_p\otimes_\mathbb{Q}\mathbb{Q}_p$ (explicitly, these maps are $x\mapsto x\otimes 1$ and $x\mapsto 1\otimes x$). While $\mathbb{Q}_p\otimes_\mathbb{Q}\mathbb{Q}_p$ is huge and monstrous and hard to understand on some level, on another level it is quite simple: it is just a tensor product of two vector spaces over $\mathbb{Q}$, and tensor products of vector spaces are easy. In particular, if $x\in\mathbb{Q}_p\setminus\mathbb{Q}$, then $1$ and $x$ are linearly independent over $\mathbb{Q}$, and so the tensors $x\otimes 1$ and $1\otimes x$ are not equal (extend $\{1,x\}$ to a basis for $\mathbb{Q}_p$ and then $x\otimes 1$ and $1\otimes x$ are two distinct elements of the induced basis for $\mathbb{Q}_p\otimes_\mathbb{Q}\mathbb{Q}_p$). So, the two inclusion maps $\mathbb{Z}_p\to\mathbb{Q}_p\otimes_\mathbb{Q}\mathbb{Q}_p$ are not equal.

(From a more abstract perspective, if you restrict to commutative algebras, you are asking whether the map $\mathbb{Z}\to\mathbb{Z}_p$ is epic in the category of commutative rings. There is a "universal" way to test whether a morphism $A\to B$ in a category (with pushouts) is epic: namely, you test whether the two inclusion maps $B\to B\coprod_A B$ into the pushout are equal (these two maps are called the "cokernel pair" of the map $A\to B$). In your case, this would mean you want to test whether the two inclusions $\mathbb{Z}_p\to\mathbb{Z}_p\otimes_\mathbb{Z}\mathbb{Z}_p$ are equal. You can do this directly, but it's slightly easier to instead shift to $\mathbb{Q}_p$ as in the second argument above since tensor products over a field are very simple.)