In which algebraic setting can I state (and prove) the binomial theorem?

I think you're looking at $$ (x+y)^n = \sum_{i=0}^n \binom ni x^i y^{n-i} $$ and thinking that the multiplication of $\binom ni$ with the other factors is the multiplication operation of the field. It's not, really; it's the "scalar" multiplication $\mathbb Z\times k\to k$ that does repeated addition. (Groups are $\mathbb Z$-modules, you see.) The binomial coefficient is really an integer.

Indeed, to prove the binomial theorem, you'd multiply out $(x+y)^n$ and then gather like terms. It's that gathering that produces the binomial coefficients — you count how many terms you have of each type. That's plain old counting, so it yields plain old integers, not elements of the field.

The binomial theorem holds, in this form, in any commutative ring. (We need multiplication to be commutative for the "like terms" to actually be like each other.) (Well, as user26857 pointed out, it's enough that $x$ and $y$ commute with each other, and the scenario where $x$ and $y$ commute but the multiplication is not commutative in general does come up pretty often.)

This is a good question. You can prove the binomial theorem (where the choose functions are interpreted as elements of the "copy" of $\mathbb{Z}$ in the ring by exactly the map you described) in any commutative ring. Induct on $n$, the case of $n = 1$ being clear. $$ (x+y)^n = (x+y)(x+y)^{n-1} = (x+y)\sum_{k=0}^{n-1}\binom{n-1}{k} x^ky^{n-1-k} $$ Expanding, the coefficient of $x^jy^{n-j}$ is $$ \binom{n-1}{j-1} + \binom{n-1}{j} $$ and we win since we are reduced to the usual statement in the integers that $$ \binom{n}{j} = \binom{n-1}{j-1} + \binom{n-1}{j} $$ (which you can prove bijectively, but which you should already accept since you're willing to accept the binomial theorem over $\mathbb{Z}$.)