$f$ don't have local minimum or local maximum in $c$

Solution 1:

If $f^{(n)}$ is continuous on $(a,b)$ this follows form Taylor's Theorem: Choose $\delta>0$ such that $f^{(n)}(x)f^{(n)}(c)> 0$ $(x \in (c-\delta,c+\delta)\subseteq (a,b))$. Let $x \in (c-\delta,c+\delta) \setminus \{c\}$. Then for some $\xi$ between $x$ and $c$ we have
$$ f(x)=\sum_{k=0}^{n-1} \frac{f^{(k)}(c)}{k!}(x-c)^k +\frac{f^{(n)}(\xi)}{n!}(x-c)^n =f(c) + \frac{f^{(n)}(\xi)}{n!}(x-c)^n. $$ If $f^{(n)}(c)> 0$, then $f^{(n)}(\xi)> 0$ (since $\xi \in (c-\delta,c+\delta)$). Since $n$ is odd $(x-c)^n > 0$ $(x\in (c,c+\delta))$ and $(x-c)^n < 0$ $(x\in (c-\delta,c))$. Thus $f(x) > f(c)$ $(x\in (c,c+\delta))$ and $f(x) < f(c)$ $(x\in (c-\delta,c))$. The case $f^{(n)}(c)< 0$ follows the same way.

If you insist that $f^{(n)}$ only exists (and is allowed to be discontinuous) then I don't know the answer.

Solution 2:

The function cannot have a local maximum or minimum in $c$, even if the last derivative is not continuous. Let us see how to approach the problem without using Taylor's Theorem, which gives you the result only in the case of continuity of the last derivative, as pointed out by Gerd.

I will change a little bit the notation, to make the process clearer.

Let $f$ be a real function defined on some neighborhood of $0$. Suppose also that the derivatives up to order $2n+1$, for some $n\geq 0$, are defined in that neighborhood. We also have that $$f^{(k)}(0) = 0, \ \ \mbox{for}\ \ k=0,1,\dots,2n$$ and that $$f^{(2n+1)}(0) > 0,$$ where $f^{(0)}(x) = f(x)$ and $f^{(k)}(x)$ is the $k$-th order derivative of $f$. Then $0$ is neither a local maximum nor a local minimum.

Note that, with respect to OP statement, but WLOG, we set $c=0$ and $f(c) = 0$.

Proof.

We have that $$\lim_{x\to 0}\frac{f^{(2n)}(x)}{x} = f^{(2n+1)}(0)>0,$$ which implies that there is a neighborhood $\mathcal I$ of $0$ such that $$\frac{f^{(2n)}(x)}{x}>0\tag{1}\label{1}$$ for all $x\in \mathcal I\setminus \{0\}$. If $n=0$, we are done, since \eqref{1} implies the thesis.

Suppose now $n>0$. For each $x\in \mathcal I\setminus \{0\}$ we must have $$f^{(2n-1)}(x) > 0\tag{2}\label{2}.$$ In fact, suppose there exists $\bar x \in \mathcal I\setminus \{0\}$ such that $f^{(2n-1)}(\bar x)\leq 0$, then by the Mean Value Theorem there would exist $\xi \in \mathcal I\setminus \{0\}$ such that $$f^{(2n)}(\xi) = \frac{f^{(2n-1)}(0)-f^{(2n-1)}(\bar x)}{0-\bar x}=\frac{f^{(2n-1)}(\bar x)}{\bar x}.$$ If $\bar x <0$, then $\bar x < \xi <0$ and $f^{(2n)}(\xi)\geq 0$. If $\bar x >0$, then $0< \xi <\bar x$, and $f^{(2n)}(\xi)\leq 0$. Both situations contradict \eqref{1}.

Equation \eqref{2} clearly implies that $f^{(2n-2)}(x)$ is monotonically increasing in $\mathcal I$. Since $f^{(2n-2)}(0)=0$ we have therefore $$\frac{f^{(2n-2)}(x)}{x}>0$$ for $x \in \mathcal I\setminus \{0\}$.

Now it is straightforward to iterate the procedure in order to show that, for $i=1,2\dots n$, $$f^{(2(n-i)+1)}(x) > 0$$ for $x \in \mathcal I\setminus \{0\}$, and $$\frac{f^{(2(n-i))}(x)}{x}>0\tag{3}\label{3}$$ for $x\in \mathcal I \setminus \{0\}$. The thesis follows immediately from \eqref{3}, with $i=n$.