What will be total number of solutions of $a+b+c = n$?

This problem is equivalent to throwing $n$ balls into three distinct pots without restrictions (first part) and with restrictions (second part). To solve part (b), first get rid of the restrictions from below, by placing $2$ balls in each pot. This leaves you with $n-6$ balls to be put in $3$ pots with max $4$ in each. Now, using inclusion-exclusion, denote by $A_i$ the set of all such placements in which there are more than $4$ balls in the $i$th pot. Then $|A_i|=\binom{n-6-5+3-1}{3-1}$, since you need to put 5 balls in the $i$th pot and then place the remaining $n-6-5$ balls into $3$ pots. Continuing this way, you'll get: $$\sum_{j=0}^3(-1)^j\binom{3}{j}\binom{n-6-5j+3-1}{3-1}=\sum_{j=0}^3(-1)^j\binom{3}{j}\binom{n-4-5j}{2}$$ The sum runs over $j$ for which the binomial coefficient makes sense, i.e. if $n<21$, then it should run up to $2$.


You want the generalized result? OK, here it is:

To find the number of integer solutions to $$a_1+a_2+\cdots+a_r=n$$ with the restrictions that $a_1$ must be in $A_1$, $a_2$ must be in $A_2$, ..., $a_r$ must be in $A_r$ (where the $A_i$ are some given sets of permissible values for the variables), you form the generating function $$f(x)=\sum_{i\in A_1}x^i\sum_{i\in A_2}x^i\cdots\sum_{i\in A_r}x^i=\sum_mc(m)x^m$$ and then the number you are looking for is $c(n)$. The tricky part is going from the product in the display to the sum, but to do that you have to know what the sets $A_1,A_2,\dots,A_r$ are and then you have to exploit whatever special properties those sets might have. For example, the sums being multiplied together may turn out to be summable as geometric series, and the binomial theorem may then be of help in expressing the product as a series.