In generatingfunctionology, for a polynomial $P$ and a differential operator $D$, what does $P(xD)$ mean?

I'm working through some of the exercises in generatingfunctionology. One of the questions is to find the generating function where the $n$th term $a_n=P(n)$ for $P$ a polynomial. The answer is $P(xD)(1/(1-x))$. I thought it meant something like if $P(n)=3n+n^2$, then $P(xD)=3xD+x^2D^2$, and then you would apply that to $1/(1-x)$. However, I know the generating function for $a_n=n^2$ is $$\frac{x(x+1)}{(1-x)^3}$$ which is not the same as $x^2D^2(1/(1-x))$. So what does it mean? Thanks.

The problem is that you've mistakenly assumed commutativity. Here $\rm\:x\:$ and $\rm\:D\:$ do not commute. Indeed $\rm\,\color{#c00}{D\:\!x = x\:\!D + 1}\,$ since $\rm\,(D\:\!x)\:\!f\:\!=\:\!D\:\!(x\:\!f)\:\!=\:\!x\:\!f\:\!'+f\:\!=\:\!(x\:\!D + 1)\:\!f.\,$ So it's not true that $\rm\:\!(x\:\!D)^2 =\:\!x^2\:\!D^2.\,$ Instead $\rm\,x\:\!\color{#c00}{D\:\!x}\:\!D\:\!=\:\!x\:\!(\color{#c00}{x\:\!D + 1})\:\!D\:\!=\:\!x^2\:\!D^2 + x\:\!D.\,$ Generally

$$\rm (x\:\!D)^n\:\!=\:\!\sum_{k\:\!=\:\!0}^n\:\!S(n,k)\:\!x^k\:\!D^k $$

where $\rm\:\!S(n,k)\:\!$ are the Stirling numbers of the second kind.

Beware $\, $ Since this operator algebra is noncommutative so many familiar polynomial identities do not hold true. For example, consider the proof of the difference of squares identity

$$\rm\:\!(x-y)\:\!(x+y)\:\!=\:\!x^2 + \color{#0a0}{x\:\!y - y\:\!x} - y^2\:\!=\:\!x^2 - y^2$$

Notice how commutativity is assumed to cancel the $\rm\color{#0a0}{middle}$ terms. Thus the identity needn't remain true if we substitute elements from some noncommutative ring, e.g. we cannot substitute $\rm\,D\,$ for $\rm\,y.\,$ Said slightly more technically, the polynomial evaluation map is a ring homomorphism iff coefficients and indeterminates commute. See this post for further remarks on this topic.

Without checking out the book, this would be my guess:

Let for instance $P(x) = x^2 + 2x$, then $P(xD) = (xD)^2 + 2xD$, which is an operator.

Let this operator work on your function $f(x) = \frac{1}{1-x}$. It's obvious that $(2xD)(f) = 2xf'$. (Here $f'$ is a bit an abuse of notation for $f'(x)$.) The other term is probably the confusing one. But if you write it as $xDxDf$ and work everything out from the right to the left you'll get

$$ xD(xf') = x(1f'+xf'') = xf' + x^2 f'' .$$

And similarly for higher order terms, so with my example polynomial $P$ we have that $P(xD)(f) = 2xf' + xf' + x^2 f''$.

So in summary your problem might be that $(xD)^2 = xDxD = xD(xD) = xD + x^2D^2 \neq x^2 D^2$, in other words $x$ and $D$ don't commute.