"Place" vs. "Prime" in a number field.

The finite places of a number field are indeed in one-to-one correspondence with the (maximal) prime ideals of its ring of integers.

Another important example is that if $C$ is a complete non-singular curve over a finite field and $k(C)$ is its function field, then the places of $k(C)$ are in one-to-one correspondence with the (closed) points of $C$.

(there's a similar statement for any field, but I don't have a reference handy to make a completely accurate statement)

In my limited experience, the point of the technical idea of a place is to give a purely field-theoretic description of this information, which can be technically much simpler to work with, and puts the finite and infinite on equal footing. It also gives a non-ad hoc way to talk about what happens "at infinity" in a number field.

The infinite places are important: in the function field case, they correspond to doing projective geometry which, in many ways, is much better behaved than affine geometry. If you're familiar with complex analysis but not algebraic geometry, the projective and affine lines (in algebraic terms) over the complex numbers are (in analytic terms) the Riemann sphere and the complex plane.

The theory of number fields is similarly improved by considering the infinite places, so it's important to consider places rather than merely primes.