Finding coefficient of generating function
Solution 1:
The original problem, after factoring out the terms as stated in the post, we have $$ x^{10}(1+x+\ldots+x^{15})\cdot x(1 + x + \ldots+x^{14})\cdot x^{20}(1+x+\ldots+x^{25}) \tag{1} $$ Applying some identities, $(1)$ gives $$ x^{31}\cdot\frac{1-x^{16}}{1-x}\cdot\frac{1-x^{15}}{1-x}\cdot\frac{1-x^{26}}{1-x} \tag{2}$$ Rearranging the terms, we then have $$ x^{31}\cdot(1-x^{16})\cdot(1-x^{15})\cdot(1-x^{26})\cdot \frac{1}{(1-x)^3} \tag{*} $$ The last term in the above product is equal to $$1 + \binom{3}{1}x + \binom{4}{2}x^2 + \ldots \tag3$$ Finally we take the products that can result in $x^{52}$ and obtain the coefficients:
- $x^{31}$ (first term in $(*)$) and $x^{21}$ (from $(3)$),
- $x^{31}$, $x^{16}$ (second term in $(*)$), $x^5$ in $(3)$
- $x^{31}$(first term in $(*)$), $x^{15}$ (third term in $(3)$), and $x^6$ in $(3)$
Since all the coefficients of the polynomials equal $1$ or $-1$ except for the polynomial expanded in $(3)$, we have as our coefficient $$ \binom{21+3-1}{21} - \binom{6+3-1}{6} - \binom{5+3-1}{5} = 204 $$
Note: I hadn't seen Andre's solution prior to typing this.
Solution 2:
Factoring out is a good beginning. Now use the fact that $1+x+x^2+\cdots+x^{n-1}=\frac{1-x^n}{1-x}$ (if $x\ne 1$).
So your expression will be a power of $x$, times a product of three polynomials of shape $1-x^k$, divided by $(1-x)^3$.
Use the generalized Binomial Theorem (or Taylor series) to find the power series for $\dfrac{1}{(1-x)^3}$. Putting the pieces together is slightly unpleasant, but doable.