Closed form for $\sum_{k=0}^{l}\binom{k}{n}\binom{k}{m}$

We present a proof of the identity by @Diger, which should be considered a starting point for additional simplification. We seek to show that

$$\sum_{k=0}^l {k\choose m} {k\choose n} = \sum_{k=0}^n (-1)^k {l+1\choose m+k+1} {l-k\choose n-k}.$$

The RHS is

$$[z^n] \sum_{k=0}^n (-1)^k {l+1\choose m+k+1} z^k (1+z)^{l-k}.$$

The coefficient extractor enforces the range:

$$[z^n] \sum_{k\ge 0} (-1)^k {l+1\choose l-m-k} z^k (1+z)^{l-k} \\ = [z^n] (1+z)^l [w^{l-m}] (1+w)^{l+1} \sum_{k\ge 0} (-1)^k w^k z^k (1+z)^{-k} \\ = [z^n] (1+z)^l [w^{l-m}] (1+w)^{l+1} \frac{1}{1+wz/(1+z)} \\ = [z^n] (1+z)^{l+1} [w^{l-m}] (1+w)^{l+1} \frac{1}{1+z+wz} \\ = [z^n] (1+z)^{l+1} [w^{l-m}] (1+w)^{l+1} \frac{1}{1+z(1+w)} \\ = [z^n] (1+z)^{l+1} [w^{l-m}] \sum_{k\ge 0} (-1)^k z^k (1+w)^{k+l+1} \\ = [z^n] (1+z)^{l+1} \sum_{k\ge 0} (-1)^k z^k {k+l+1\choose l-m}.$$

This is

$$\bbox[5px,border:2px solid #00A000]{ \sum_{k=0}^n (-1)^k {l+1\choose n-k} {k+l+1\choose l-m}.}$$

The LHS is

$$\sum_{k\ge 0} [[0\le k\le l]] [z^m] (1+z)^k [w^n] (1+w)^k \\ = [z^m] [w^n] \sum_{k\ge 0} (1+z)^k (1+w)^k [v^l] \frac{v^k}{1-v} \\ = [z^m] [w^n] [v^l] \frac{1}{1-v} \sum_{k\ge 0} (1+z)^k (1+w)^k v^k \\ = [z^m] [w^n] [v^l] \frac{1}{1-v} \frac{1}{1-(1+z)(1+w)v} \\ = [z^m] [w^n] [v^l] \frac{1}{v-1} \frac{1/(1+z)/(1+w)}{v-1/(1+z)/(1+w)}.$$

The inner term is

$$\mathrm{Res}_{v=0} \frac{1}{v^{l+1}} \frac{1}{v-1} \frac{1/(1+z)/(1+w)}{v-1/(1+z)/(1+w)}.$$

Residues sum to zero and the residue at infinity in $v$ is zero. The contribution from minus the residue at $v=1/(1+z)/(1+w)$ is

$$- [z^m] (1+z)^{l+1} [w^n] (1+w)^{l+1} \frac{1/(1+z)/(1+w)}{1/(1+z)/(1+w)-1} \\ = - [z^m] (1+z)^{l+1} [w^n] (1+w)^{l+1} \frac{1/(1+z)}{1/(1+z)-(1+w)} \\ = [z^m] (1+z)^{l+1} [w^n] (1+w)^{l+1} \frac{1/(1+z)}{w+z/(1+z)} \\ = [z^m] (1+z)^{l+1} [w^n] (1+w)^{l+1} \frac{1/z}{w(1+z)/z+1}.$$

Now with $l,m,n$ positive integers we must have $l\ge n,m$ or else there is no contribution to $k^\underline{m} k^\underline{n}.$ This means we continue with

$$[z^m] (1+z)^{l+1} \sum_{k=0}^n {l+1\choose k} \frac{1}{z} (-1)^{n-k} \frac{(1+z)^{n-k}}{z^{n-k}} \\ = \sum_{k=0}^n (-1)^{n-k} {l+1\choose k} {l+1+n-k\choose m+1+n-k}.$$

This is $$\bbox[5px,border:2px solid #00A000]{ \sum_{k=0}^n (-1)^{n-k} {l+1\choose k} {l+1+n-k\choose l-m}.}$$

We have the same closed form for LHS and RHS, thus proving the claim.

For a full proof we also need to show that the contribution from $v=1$ is zero. We get

$$[z^m] [w^n] \frac{1/(1+z)/(1+w)}{1-1/(1+z)/(1+w)} = [z^m] [w^n] \frac{1}{(1+z)(1+w)-1} \\ = [z^m] [w^n] \frac{1}{z+w+zw} = [z^{m+1}] [w^n] \frac{1}{1+w(1+z)/z} \\ = [z^{m+1}] (-1)^n \frac{(1+z)^n}{z^n} = (-1)^n {n\choose n+m+1} = 0.$$

I doubt there is closed form, but this is another identity which can be derived by contour integration $$\sum_{k=0}^l {k \choose m} {k \choose n} = \sum_{k=0}^n (-1)^k {l+1 \choose m+k+1}{l-k \choose n-k} \, .$$ If you are interested I can write it down. It is useful when $l$ is large and either $m$ or $n$ is small.

edit: On part of your try the third row is still correct, while the fourth equality (first time no sum) is wrong.