Ring of the integer $p$-adic numbers $\mathbb{Z}_p$

Let's look at a different ring first: $R=k[[X]]$, the ring of formal power series over a field. We can also think of $R$ as the ring of sequences $f_0, f_1, f_2,\ldots$, where each $f_i$ is a polynomial of degree $\leq i$, and $f_i = f_{i+1} \pmod {X^{i+1}}$ for each $i$.

In other words, we construct a power series by choosing polynomials of higher and higher degree, and doing so in a consistent way. Yet another way to write this is $\displaystyle R = \varprojlim_{i\geq 0} k[X]/(X^i)$

(This last way emphasizes that $k[[X]]$ is the completion of $k[X]$ with respect to the $(X)$-adic topology, if you are interested in the general form of this construction. Completions are very important throughout algebra!)

Easy fact: $(X^i)$ is an ideal of $R$, and $\bigcap_{i\geq 0} (X^i) = (0)$. Intuitively, this tells us that the sequence $(X),(X^2),(X^3),\ldots$ "approaches zero", that is, we can evaluate how "close" two power series $f$ and $g$ are by seeing how many of their leading terms are the same. If $f-g \in (X^6)$, that's "closer" than $f-g\in (X^5)$. If $f-g\in (X^i)$ for all $i$, then $f=g$.

Another easy fact: Every polynomial is also a power series! In other words, there is a natural injective homomorphism $k[X] \to k[[X]]$. It is not hard to check that this map gives an isomorphism $k[X]/(X^i) \cong k[[X]]/(X^i)$ for every $i\geq 0$. (Intuitively: a truncated power series is no different from a truncated polynomial.)

Harder fact: Any power series with nonzero constant term is invertible. To see this, suppose that $f=a+gX$, with $a\neq 0$. By scaling, we may assume that $a=1$, so $f=1+gX$.

We have $(1+gX)(1-gX) = 1-g^2 X^2$, which is "close" to $1$. And $(1+gX)(1-gX+g^2X^2) = 1-g^3 X^3$, which is even closer to $1$. And just to make the point, $(1+gX)(1-gX+g^2X^2 - g^3 X^3) = 1-g^4 X^4$, closer still.

So we define $h$ to be the power series $1-gX+g^2X^2-g^3X^3 + \cdots$. Take a minute to verify that we are allowed to do this. The key is that, for any $i$, we have an explicit formula for $h \pmod{X^i}$, and these formulas are consistent with each other.

Then $fh = 1-g^iX^i = 1 \pmod{X^i}$. Because this holds for all $i$, $fh=1$. So $f$ is invertible.

By the way, this implies that any nonzero power series can be written uniquely in the form $uX^n$, with $u\in R^\times$. (Proof: Divide by the leading term!) This observation will be important later.

Intuition: The above has an interpretation in the domain of complex analysis: If $f$ is an analytic function on an open set containing $0\in\mathbb{C}$, then either $f\equiv 0$, or $f$ vanishes at $0$ with order $n\geq 0$, and $f/X^n$ is invertible (i.e. nonvanishing) in a neighborhood of $0$.

Here is a very useful general fact about rings: if $P\subsetneq R$ is a proper ideal, and every element outside of $P$ is invertible, then $P$ is the unique maximal ideal of $R$. In fact, the statement $R\setminus P = R^\times$ is equivalent to the statement that $P$ is the unique maximal ideal.

Why is this? Well, if $I\subsetneq R$ is any proper ideal, then $I\cap R^\times = \emptyset$. This implies that $R\setminus P = R^\times \subset R\setminus I$. In other words $I\subset P$. So every ideal of $R$, besides $R$ itself, is contined in $P$!

Applied to the setting of power series, we see that $f\notin (X) \implies f\in R^\times$. Therefore, $(X)$ is the unique maximal ideal of $R$.

Next, let's determine all the ideals of $R$. Specifically, let's show that the only nonzero proper ideals are the ones we already know about: $(X), (X^2), \cdots$.

Let $I$ be a nonzero proper ideal, so that $I\subset (X)$. If $I\subset (X^i)$ for all $i$, then $I=(0)$, so choose the smallest $n$ such that $I\subset (X^n)$, but $I\not\subset (X^{n+1})$.

Pick some $f\in I$ with $f\notin (X^{n+1})$. We have $f\in (X^n)$ however, so we can write $f=g X^n$, where $g\notin (X)$. But then $g$ is invertible, so $fg^{-1} = X^n \in I$. This implies that $(X^n)\subset I$. By assumption, $I\subset (X^n)$, so $I=(X^n)$.

Note that we can now conclude very easily that $R$ is a principal ideal domain, because we know all the ideals, and they're all principal!

Going down your list: We have just established, for $k[[X]]$, every property that you were interested in for $\mathbb{Z}_p$: We characterized the units, we established that every ideal is a power of the maximal ideal, we described factorization, we pointed out that it's a PID, and we mentioned the inclusion $k[X] \to k[[X]]$ and its nice properties.

Every single thing about the above explanation still holds if we replace $k[X]$ by $\mathbb{Z}$, $(X)$ by $(p)$, and $k[[X]]$ by $\mathbb{Z}_p$. All the proofs still work, though they may be a little less intuitive in some places.

In fact, the integers are "almost" a polynomial ring $\mathbb{Z}/p [p]$, in the sense that every nonnegative integer has a representation $k = a_0 + a_1 p + \cdots +a_n p^n$, and this representation is unique if we add the restriction $a_i \in \{0,1,\cdots , p-1\}$.

In other words, the point is to think of the base-$p$ representation as something like a polynomial in $p$. If we pursue this metaphor, $\mathbb{Z}_p$ is something like a power series in $p$—specifically, it's just the collection of base-$p$ representations, but where we allow infinitely many digits.

The only thing lacking from the previous section is: What happens to negative integers? While nonnegative integers look quite ordinary in $\mathbb{Z}_p$, we should be careful to see what happens to the negative ones.

$-1$ is, of course, not a positive integer. But watch this: $-1 \equiv p-1 \pmod{p}$, $-1 \equiv (p-1)(1+p)\pmod{p^2}$, $-1 \equiv (p-1)(1+p+p^2)\pmod{p^2}$, et cetera.

So, $-1$ has a base $p$ representation (albeit one with infinitely many terms) given by $-1 = (p-1) + (p-1)p + (p-1)p^2 + (p-1)p^3 + \cdots$. While your teachers would normally yell at you for summing a divergent series like this, it is a perfectly valid statement in $\mathbb{Z}_p$, and it completes our picture of the embedding $\mathbb{Z}\to\mathbb{Z}_p$.

This may be the least part of the explanation, but in principle it is exactly the same as the identity $0.99999\cdots = 1$, which we can verify by calculating $1-0.99999\cdots$. Similarly, we can verify that $-1 = (p-1) + (p-1)p + (p-1)p^2 + (p-1)p^3 + \cdots$ by calculating $1+ (p-1) + (p-1)p + (p-1)p^2 + (p-1)p^3 + \cdots = p + (p-1)p + (p-1)p^2 + (p-1)p^3 + \cdots = p^2 + (p-1)p^2 + (p-1)p^3 = p^3 + (p-1)p^3 + \cdots = \cdots$, which visibly lies in $\bigcap_{i\geq 0} (p^i) = (0)$!

By the way, $\mathbb{Z}_p$ is often described as the metric completion of $\mathbb{Z}$ with respect to the $p$-adic norm $\| a\| = p^{-\nu_p (a)}$ (e.g. $\| p\| = 1/p$, $\|k\| = 1$ when $(k,p)=1$).

If you squint closely, these two definitions are the same: we are saying that two integers $a$ and $b$ are "close" when $a-b$ is divisible by lots of powers of $p$, which is somehow the same as the property $\bigcap_{i\geq 0} (p^i) = (0)$.