Functions satisfying $\sum_{n=0}^k(-1)^n\binom{k}{k-n}f^{k-n}(x)=0$.

I first tried to modify the proof at the link in the question, but soon I realized that the proof holds because of the particular form satisfied in the case $n = 2$.

Instead, I was able to deduce a partial answer:

Proposition. Let assume that a continuous function $f : \Bbb{R} \to \Bbb{R}$ satisfies $$ \sum_{k=0}^{n} \binom{n}{k} (-1)^{n-k} f^k (x) = a, \quad \forall x \in \Bbb{R} \tag{1}$$ for some real number $a \in \Bbb{R}$ and some integer $n \geq 1$. If we further assume that $f$ has at least one fixed point, then $f$ is the identity function.

We first observe the following properties.

Lemma 1. Let $f$ be a continuous function satisfying $(1)$. That is, let $f$ satisfy the assumption of Proposition except that $f$ has a fixed point. Then $f$ is an increasing bijection from $\Bbb{R}$ onto $\Bbb{R}$.

Proof. Assume $f(x) = f(y)$. Then

$$ x = (-1)^n a - \sum_{k=1}^{n} \binom{n}{k} (-1)^{k} f^k (x) = (-1)^n a - \sum_{k=1}^{n} \binom{n}{k} (-1)^{k} f^k (y) = y $$

and $f$ is injective. In particular, $f(\Bbb{R})$ is an open interval.

To prove the surjectivity of $f$, assume that $f$ is not surjective. Then either $\sup f$ or $\inf f$ is finite, and in any cases we can find a real number $\alpha \in \Bbb{R}$ and a sequence of real numbers $x_j$ such that $|x_j| \to \infty$ and $f(x_j) \to \alpha$ as $j \to \infty$. But this implies

\begin{align*} \infty = \lim_{j\to\infty} \left| x_j \right| = \lim_{j\to\infty} \left| (-1)^n a - \sum_{k=1}^{n} \binom{n}{k} (-1)^{k} f^k (x_j) \right| = \left| (-1)^n a - \sum_{k=1}^{n} \binom{n}{k} (-1)^{k} f^{k-1}(\alpha) \right| < \infty, \end{align*}

a contradiction. Therefore $f$ is surjective. In particular, $f$ is a homeomorphism and it is either increasing or decreasing.

To prove that $f$ is increasing, assume that $f$ is decreasing. Then $(-1)^{k-1} f^{k}$ is decreasing for any $k \geq 1$. Thus the right-hand side of

$$ x = (-1)^n a + \sum_{k=1}^{n} \binom{n}{k} (-1)^{k-1} f^k (x) $$

is also decreasing, a contradiction. This completes the proof. ■

Lemma 2. Suppose that $f : \Bbb{R} \to \Bbb{R}$ be an continuous increasing bijection, so that it extends to a continuous bijection $\bar{f}$ from $\bar{\Bbb{R}} = [-\infty, \infty]$ onto $\bar{\Bbb{R}}$. Then for any $x \in \bar{\Bbb{R}}$, we have the following trichotomy:

1. If $x < f(x)$, then $f^k (x)$ converges to the smallest fixed point of $\bar{f}$ greater than $x$.
2. If $x > f(x)$, then $f^k (x)$ converges to the greatest fixed point of $\bar{f}$ smaller than $x$.
3. If $x = f(x)$, then $f^k (x)$ converges to $x$.

Proof. Since $\bar{f} : \bar{\Bbb{R}} \to \bar{\Bbb{R}}$ is continuous, the set $F = \{ \bar{f}(x) = x : x \in \bar{\Bbb{R}} \}$ is a closed set containing both $\infty$ and $-\infty$. If $F = \bar{\Bbb{R}}$, there is nothing to prove and assume that $F^c \neq \varnothing$. Since $F^c$ is open in $\Bbb{R}$, we can decompose it as a countable union of disjoint open intervals $U_j$. Note that on each $U_j$, either $f(x) > x$ identically for all $x \in U_j$ or $f(x) < x$ identically for all $x \in U_j$ by continuity of $f$.

Now let $x \neq f(x)$. Then $x \in U_j$ for some $j$. Assume first that $x < f(x)$. Then for any $x < y \leq f(x)$, we have $f(y) > f(x) \geq y$ and we have $[x, f(x)] \subset U_j$. In particular, $f(x) \in U_j$. Now this argument can be repeatedly applied to yield that $f^k (x) \in U_j$ for all $k$. Since $f^k(x)$ is monotone increasing, it must converge to some point $\alpha \in \bar{\Bbb{R}}$. It is plain to check that $\alpha = \sup U_j$ and $\alpha$ is a fixed point of $\bar{f}$, proving the first option of the trichotomy. The second option follows exactly the same manner and the third option is trivial. ■

As an immediate consequence, we obtain the following corollary:

Corollary. If $f : \Bbb{R} \to \Bbb{R}$ is a continuous increasing bijection having at least one fixed point, then for any $x \in \Bbb{R}$, either $f^k (x)$ or $f^{-k} (x)$ converges to a fixed point of $f$.

Proof. Let $U_j$ be as in the proof above. If $x \in U_j$, then $f^k (x)$ and $f^{-k} (x)$ converges to $\sup U_j$ and $\inf U_j$. Since $f$ has a fixed point, either $\sup U_j$ or $\inf U_j$ is finite. ■

Now we are ready to prove the proposition.

Proof of Proposition. Define

$$g(x) = \sum_{k=0}^{n-1} \binom{n-1}{k} (-1)^{n-1-k} f^k (x). $$

It is plain to observe that $g \circ f - g = a$. Let $F$ be the set of fixed points of $f$. Then $F$ is non-empty by assumption. If we pick a point $\alpha \in F$, then

$$0 = g(f(\alpha)) - g(\alpha) = a.$$

In particular, we have

$$ g = g\circ f = g \circ f^{k} \quad \forall k \in \Bbb{Z}. $$

Thus if $n = 1$, then $g(x) = x$ and this identity immediately yields $f(x) = x$. This shows that we may assume $n \geq 2$.

In view of the corollary above, for each $x \in \Bbb{R}$, either $f^{k}(x)$ or $f^{-k}(x)$ converges to a point of $F$. Thus there exists $\alpha \in F$ such that

$$g(x) = g(\alpha) = 0,$$

where the last inequality holds by the definition of $g$ together with the assumption $n \geq 2$. Putting together, we showed that if $(1)$ holds for some $n \geq 2$ and $a \in \Bbb{R}$, then $(1)$ also holds for $n-1$ instead of $n$ and $a = 0$. Therefore by the mathematical induction, we obtain the desired conclusion. ■

Let $P\in\Bbb R[X]$ be a polynomial of degree${}<k$, such that its polynomial function $p:\Bbb R\to\Bbb R$ is nowhere decreasing (in other words $p'(x)\geq0$ for all $x\in\Bbb R$; this obviously requires $\deg P$ to be odd). Then $p^{-1}$ is well defined and continuous. Define $f:x\mapsto p(p^{-1}(x)+1)$ which is also continuous, and satisfies $f^n(p(x))=p(x+n)$ for all $n$. Now $$ \sum_{i=0}^k(-1)^i\binom k{k-i}f^{k-i}(p(x)) =\sum_{i=0}^k(-1)^i\binom kip(x+k-i) =\Delta^k(p)(x+k) =0 $$ for all $x\in\Bbb R$, where $\Delta$ is a finite difference operator defined by $\Delta g: x\mapsto g(x)-g(x-1)$, which decreases the degree of polynomial functions, so that $\Delta^k(p)=0$ identically.

Since $p$ is surjective this shows $f$ satisfies your requirement and that the answer to your first question is negative for $k\geq4$. The simplest counterexample is $f:x\mapsto (\sqrt[3]x+1)^3$.