How is Faulhaber's formula derived?

I have been wanting to understand how to find the sum of this series.

$$1^p + 2^p + 3^p +{\dots} + n^p$$

I am familiar with Gauss' diagonalised adding trick for the sum of the first $n$ natural numbers.

I can prove the formulas for

$$\begin{align} \sum_{1}^{n} k^2 &= \frac{n(n+1)(2n+1)}{6}\\ \sum_{1}^{n} k^3 &= \frac{n^2(n+1)^2}{4}\\ \sum_{1}^{n} k^4 &= \frac{n(n+1)(2n+1)(3n^2+3n-1)}{30} \\ \end{align}$$

With mathematical induction. But, beyond that even proofs with mathematical induction are difficult.

I'm interested in learning the theory and the proof behind Faulhaber's formula. What is the knowledge required to understand this proof ?

The following interesting derivation is from Aigner's "A Course in Enumeration" (Springer, 2007).

Remember that if we have the following exponential generating functions:

$\begin{align} \widehat{A}(z) &= \sum_{n \ge 0} a_n \frac{z^n}{n!} \\ \widehat{B}(z) &= \sum_{n \ge 0} b_n \frac{z^n}{n!} \end{align}$

then also:

$\begin{align} \widehat{A}(z) \cdot \widehat{B}(z) &= \sum_{n \ge 0} \left( \sum_{0 \le k \le n} \frac{a_k}{k!} \frac{b_{n - k}}{(n - k)!} \right) z^n \\ &= \sum_{n \ge 0} \left( \sum_{0 \le k \le n} \frac{n!}{k! (n - k)!} a_k b_{n - k} \right) \frac{z^n}{n!} \\ &= \sum_{n \ge 0} \left( \sum_{0 \le k \le n} \binom{n}{k} a_k b_{n - k} \right) \frac{z^n}{n!} \end{align}$

Let's define:

$\begin{align} S_m(n) = \sum_{1 \le k \le n - 1} k^m \end{align}$

We can define the exponential generating function:

$\begin{align} \widehat{S}_n(z) &= \sum_{m \ge 0} S_m(n) \frac{z^m}{m!} \\ &= \sum_{1 \le k \le n - 1} \sum_{m \ge 0} \frac{k^m z^m}{m!} \\ &= \sum_{1 \le k \le n - 1} \mathrm{e}^{k z} \\ &= \frac{\mathrm{e}^{n z} - 1}{\mathrm{e}^z - 1} \\ \end{align}$

This is almost the exponential generating function of the powers of $n$:

$\begin{align} \widehat{P}_n(z) &= \sum_{m \ge 0} n^m \frac{z^m}{m!} \\ &= \mathrm{e}^{n z} \end{align}$

We can write:

$\begin{align} (\widehat{P}_n(z) - 1) \widehat{B}(z) = z \widehat{S}_n(z) \tag{1} \end{align}$

where we have the exponential generating function of the Bernoulli numbers:

$\begin{align} \widehat{B}(z) &= \frac{z}{\mathrm{e}^z - 1} \\ &= \sum_{n \ge 0} B_n \frac{z^n}{n!} \end{align}$

Comparing coefficients of $z^{m + 1}$ in (1):

$\begin{align} \sum_{m \ge 1} z^m \sum_{0 \le k \le m} \binom{m}{k} \frac{(n z)^{m - k}}{(m - k)!} B_k = \sum_{m \ge 0} S_m(n) \frac{z^{m + 1}}{m!} \end{align}$

we get after simpĺifying:

$\begin{align} S_m(n) = \frac{1}{m + 1} \, \sum_{0 \le k \le m} \binom{m + 1}{k} B_k n^{m + 1 - k} \end{align}$

Note: The formula given is often associated with Faulhaber, but Faulhaber's formulas where quite different (and computationally more efficient). This formula is due to Bernoulli.

For the expression of $\sum n^k$ as a polynomial in $n$ with Bernoulli numbers in the coefficients, the background is calculus of finite differences.

For seeing why and when the linear factors $n$, $n+1$ and $2n+1$ divide the answer, the background needed is some basic algebra of polynomials. Also for showing that the sum for odd $k$ is a polynomial in $n(n+1)$.

The Wikipedia page is informative and links to an expository paper by Beardon on the same subject.

This probably won't take you all the way to Faulhaber's formula, but it can help you get to the closed form of $\sum_{k=1}^nk^p$ for any whole $p$.

$$\sum_{k_1=0}^{n-1}\sum_{k_2=0}^{k_0-1}\sum_{k_3=0}^{k_1-1}\dots\sum_{k_x=0}^{k_{x-1}-1}1=\frac{n(n-1)(n-2)\dots(n-x+1)}{1\times2\times3\times\dots\times x}$$

$$=\frac{n!}{x!(n-x)!}=\binom nx$$

For example,

$$\sum_{k=1}^nk^0=1+1+1+\dots+1=\sum_{k=0}^{n-1}1=n$$

$$0^1+\sum_{k=1}^nk^1=\sum_{k=0}^nk^1=\sum_{k_1=0}^{(n+1)-1}\sum_{k_2=0}^{k_1-1}1=\frac{(n+1)((n+1)-1)}2=\frac{n(n+1)}2$$

And you can keep doing this for higher values of $p$.