A continuous function that attains neither its minimum nor its maximum at any open interval is monotone

Solution 1:

This fact is actually true even if $f$ is not continuous but attains its maximum and minimum in every closed interval (for example, $f(x)=x+\operatorname{sgn}x$).

First, notice that $f$ is injective: If we could find $x_1<x_2$ such that $f(x_1)=f(x_2)$, then $f$ would necessarily attain both its maximum and minimum in the open interval $(x_1,x_2)$.

Let $a<b<c$ in $\mathbb{R}$ be arbitrary. Let $M=\max f[a,c]$ and $m=\min f[a,c]$. Since $f$ does not attains it maximum nor its minimum in the open interval $(a,c)$, then either $(f(a),f(c))=(m,M)$ or $(f(a),f(c))=(M,m)$.

Now, let's show that the following are equivalent:

(i) $f(a)<f(b)$.

(ii) $f(b)<f(c)$.

(iii) $f(a)<f(c)$.

Proof: (i)$\Rightarrow$(ii). Then $f(a)<f(b)\leq M$, so $f(a)\neq M$, thus $f(a)=m$ and $f(c)=M>f(b)$.

(ii)$\Rightarrow$(iii). Then $m\leq f(b)<f(c)$, so $f(c)\neq m$, thus $f(c)=M$ and $f(a)=m<M=f(c)$.

(iii)$\Rightarrow$(i). In this case, necessarily $f(a)=m$ and $f(c)=M$, so in particular $f(a)=m\leq f(b)$, and, since $f$ is injective, $f(a)\neq f(b)$, so $f(a)<f(b)$.

Finally, by changing $f$ by $-f$ is necessary, we may suppose that $f(0)<f(1)$. Let's show that $f$ is (strictly) increasing: Let $x<y$ in $\mathbb{R}$ be arbitrary. Let $K=|x|+|y|+1$. Notice that $-K<x<y<K$.

Using $a=0$, $b=1$, $c=K$ and $(i)\Rightarrow(iii)$ as above, we obtain $f(0)<f(K)$.

Using $a=-K$, $b=0$, $c=K$ and $(ii)\Rightarrow(iii)$ as above, we obtain $f(-K)<f(K)$.

Using $a=-K$, $b=x$, $c=K$ and $(iii)\Rightarrow (i)$ as above, we obtain $f(-K)<f(x)$.

Finally, using $a=-K$, $b=x$, $c=y$ and $(i)\Rightarrow(ii)$ as above, we obtain $f(x)<f(y)$.

Therefore, $f$ is strictly increasing, hence monotone.