Number of ways to put $n$ unlabeled balls in $k$ bins with a max of $m$ balls in each bin
Solution 1:
As a check I did it with an inclusion-exclusion argument, getting $$\sum_i(-1)^i\binom{k}i\binom{n+k-1-i(m+1)}{k-1}\,.$$
Setting $r=i$ and $r_2=n-i(m+1)$ to match your notation, I make this $$\sum_r (-1)^r\binom{k}r\binom{k+r_2-1}{r_2}\,.$$ It appears that you’ve the wrong exponent on $-1$.
Solution 2:
Let's denote the problem as $D(n,k,m)$, then when $mk-n \leq m$, the answer can be given as: $D(n,k,m)=\binom{km-n+k-1}{k-1}$, where $m \leq n$.
It can be easily verified using $D(5,2,3)=2$, or $D(6,3,3)=10$, etc..
Let me try to explain it in more details with my poor English (sorry for that).
Suppose we start from the state that all bins are filled with $m$ balls in each. Then the task for us is to eliminate $mk-n$ balls from these $k$ bins. To ditribute the $mk-n$ "elimination" into $k$ bins, we have $\binom{km-n+k-1}{k-1}$ different ways if $mk-n \leq m$, i.e. the number of elimination in each bin will not exceed $m$.
Solution 3:
The number of ways to put $n$ balls in $k$ boxes with in each box a maximum of $m$ balls is the coefficient of $[q^n]$ in the $\text{QBinomial}(k+m,k,q)$.
For example, the number of different ways to put $n = 5$ balls in $k = 4$ boxes with in each box no more than $m = 3$ balls is given by
$$
\text{QBinomial}(7,4,q) = (q^6+q^5+q^4+q^3+q^2+q+1)(q^2-q+1)(q^4+q^3+q^2+q+1) =
q^{12}+q^{11}+2q^{10}+3q^9+4q^8+4q^7+5q^6+4q^5+4q^4+3q^3+2q^2+q+1.
$$
The coefficient of $q^5$ is $4$. Hence there are four ways. These four ways are $(2,3)$, $(1,1,3)$, $(1,2,2)$, $(1,1,1,2)$.