Solving 9 sons puzzle

In your equation, if you solve for $a$ you get:

$$a = \frac{\sqrt{M^2-60b^2}}{3} - 4b$$ If we're looking for integer solutions then this means that $M^2-60b^2$ must be a square. Also it must be divisible by $3$, so at least you only need search over (pseudo-)triples of the form: $$9x^2 + 60b^2 = M^2$$ Moreover you want $a$ to be positive so you require $x \ge 4b$. There is only one primitive triple with this property which yields $\{a,b,M\} = \{2,3,48\}$. Any multiple of this triple would also work ($\{4,6,96\}, \{6,9,144\}, \ldots$).

Idea:

You can simplify things a bit if you make the middle kid be $n$ years old with a difference of $k$. Then the kids ages are $n-4k, n-3k, n-2k, n-k, n, n+k, n+2k, n+3k, n+4k$. The sum of the squares of the ages is $9n^2+60k^2$.

Dad's age is $\sqrt{9n^2+60k^2}$.

I think you can find reasonable values fairly quickly that give a realistic integer-valued solution.

All the coefficients on the left are multiples of $3$, so $M$ must be a multiple of $3$, let $M=3N$. Now we have $a^2+8ab+\frac {68}3b^2=N^2$and $b$ must be a multiple of $3$. It is pretty unreasonable for $b$ to equal $6$, as the sons would span $48$ years, but you can try it if you want. We have still cut down the case work. Nothing more pops out at me. At this point I would just make a spreadsheet with possible values for $a$ and find which one gives a square for $N^2$

The discriminat of the equation $$9a^2+72ab+204b^2-M^2=0$$ is $$4(M^2-540b^2)$$ so $M^2-540b^2$ must be a perfect square, say, $n^2$. Therefore, $$540b^2=M^2-n^2=(M+n)(M-n)$$

The arithmetic mean of $M+n$ and $M-n$ is $M$, and it is greater that the geometric mean, $\sqrt{540b^2}>23b$. Since $M$ is the age of a man, $b\le 5$.

Since $M+n$ and $M-n$ have the same parity, both must be even, so we can write $$135b^2=\frac{M+n}2\cdot\frac{M-n}2$$ and $\dfrac{M-n}2$ is a divisor of $135b^2$ lesser than $b\sqrt{135}$.