Proving identity $ \binom{n}{k} = (-1)^k \binom{k-n-1}{k} $. How to interpret factorials and binomial coefficients with negative integers.

Solution 1:

The given formula clearly assumes that binomial coefficients $\binom nk$ are defined at least for all $n\in\Bbb Z$ (but one may stick to $k\in\Bbb N$): the only instance of the formula that avoids negative upper indices is when $0\leq n<k$ but then it gives the entirely uninteresting identity $0=0$. The usual combinatorial interpretation of$~\binom nk$ can of course not be applied when $n<0$, so something else must guide extending the definition to such cases.

It is not a priori clear that one should try to extend the definition at all, but it turns out to be quite useful. Here are some reasons. One observation is that for fixed $k$, the value $\binom nk$ is a degree $k$ polynomial function of $n$, and this suggests that one use this as a definition for cases where $n\notin\Bbb N$. Indeed with $$ \binom Xk \overset {\rm def} = \frac{X(X-1)\ldots(X-k+1)}{k!}\in\Bbb Q[X] \qquad\text{for any $k\in\Bbb N$}\tag1 $$ we get the usual binomial coefficients at $X:=n\in\Bbb N$, and one can define $\binom xk$ for almost any value $x$ by substitution into the polynomial $\binom Xk$. This works notably for negative integers, which is the most important extension case, but it also works for rational or even complex numbers $x$ (algebraically the only restriction is that $x$ must be in a ring containing $\Bbb Q$; this allows much freedom, but excludes most notably elements of $\Bbb Z/n\Bbb Z$). Writing out $\binom{-X}k$ gives $\binom{-X}k=(-1)^k\binom{X+k-1}k$, so the formula of the question indeed becomes a fundamental identity. Another fundamental identity is Pascal's recurrence $$ \binom{X+1}k=\binom X{k-1}+\binom Xk \qquad\text{for integer $k>0$}\tag2 $$ which is easily checked. It should be noted by contrast that symmetry $\binom nk=\binom n{n-k}$ is no longer meaningful or true when $n\in\Bbb N$ is replaced by a general $x$.

The main fact that makes the newly defined values $\binom xk$ useful is the formal power series identity $$ (1+X)^\alpha = \sum_{k\in\Bbb N}\binom\alpha kX^k \tag 3 $$ that holds for them. I should say it is initially only an identity for $\alpha\in\Bbb Z$, when the left hand side is unambiguously defined as formal power series; for $\alpha\geq0$ it is the binomial formula (with $Y=1)$, while for $\alpha<0$ it can be proved by induction on $-\alpha$ using $(2)$. However since the additive law for exponents can also be proved, the RHS of $(3)$ provides the canonical choice for fractional powers of $(1+X)$ as power series, and in fact serves as definition of $(1+X)^\alpha$ for arbitrary $\alpha$.

The idea chosen in the question to write binomial coefficients in terms of factorials and then to replaces these by instances of the Gamma function is rather unfruitful. It leads to definitional complications (because the Gamma function is undefined) exactly for the case of negative integer upper index, which is of most direct interest.

Solution 2:

In this answer, I show that $$ \begin{align} \binom{-n}{k} &=\frac{\overbrace{-n(-n-1)(-n-2)\dots(-n-k+1)}^{k\text{ factors}}}{k!}\\ &=(-1)^k\frac{(n+k-1)(n+k-2)(n+k-3)\dots n}{k!}\\ &=(-1)^k\binom{n+k-1}{k} \end{align} $$

Solution 3:

When $a$ is not a nonnegative integer, $\binom{a}{k}$ is defined by $$\binom{a}{k} = \frac{a(a-1)\cdots(a-k+1)}{k!},$$ where the numerator has $k$ terms.

Using this definition you should be able to prove your result.