Uses of $\lim \limits_{h\to 0} \frac{f(x+h)-f(x-h)}{2h}$

The "symmetric difference" form of the derivative is quite convenient for the purposes of numerical computation; to wit, note that the symmetric difference can be expanded in this way:

$$D_h f(x)=\frac{f(x+h)-f(x-h)}{2h}=f^\prime(x)+\frac{f^{\prime\prime\prime}(x)}{3!}h^2+\frac{f^{(5)}(x)}{5!}h^4+\dots$$

and one thing that should be noted here is that in this series expansion, only even powers of $h$ show up.

Consider the corresponding expansion when $h$ is halved:

$$D_{h/2} f(x)=\frac{f(x+h/2)-f(x-h/2)}{h}=f^\prime(x)+\frac{f^{\prime\prime\prime}(x)}{3!}\left(\frac{h}{2}\right)^2+\frac{f^{(5)}(x)}{5!}\left(\frac{h}{2}\right)^4+\dots$$

One could take a particular linear combination of this half-$h$ expansion and the previous expansion in $h$ such that the term with $h^2$ zeroes out:

$$4D_{h/2} f(x)-D_h f(x)=3f^\prime(x)-\frac{f^{(5)}(x)}{160}h^4+\dots$$

and we have after a division by $3$:

$$\frac{4D_{h/2} f(x)-D_h f(x)}{3}=f^\prime(x)-\frac{f^{(5)}(x)}{480}h^4+\dots$$

Note that the surviving terms after $f^\prime(x)$ are (supposed to be) much smaller than either of the terms after $f^\prime(x)$ in the expansions for $D_h f(x)$ and $D_{h/2} f(x)$. Numerically speaking, one could obtain a slightly more accurate estimate of the derivative by evaluating the symmetric difference at a certain (well-chosen) step size $h$ and at half of the given $h$, and computing the linear combination $\dfrac{4D_{h/2} f(x)-D_h f(x)}{3}$. (This is akin to deriving Simpson's rule from the trapezoidal rule). The procedure generalizes, as one keeps taking appropriate linear combinations of a symmetric difference for some $h$ and the symmetric difference at half $h$ to zero out successive powers of $h^2$; this is the famous Richardson extrapolation.

Lemma: Let $f$ be a convex function on an open interval $I$. For all $x \in I$, $$ g(x) = \lim_{h \to 0} \frac{f(x+h) - f(x-h)}{2h} $$ exists and $f(y) \geq f(x) + g(x) (y-x)$ for all $y \in I$.

In particular, $g$ is a subderivative of $f$.

$$ \begin{eqnarray*} \lim_{h\to 0} \frac{f(x+h)-f(x-h)}{2h} &=& \frac12 \lim_{h\to 0}\left(\frac{f(x+h)-f(x)}h+\frac{f(x)-f(x-h)}h\right) \\ &=& \frac12 (f'(x)+f'(x)) = f'(x) \end{eqnarray*} $$

Assuming, of course that $f$ is differentiable at $x$.

This cannot be used as a definition of the derivative. First the result is half the sum of the left and right derivatives at $x$, when these exist. Second the limit can be well defined even when the sided derivatives do not exist, consider for example $f(x)=|x|^a$ around $x=0$ for suitable values of $a$. More generally, the limit at $x$ exists and is $g'(x)$ as soon as $f=g+s$ with $g$ differentiable at $x$ and $s$ symmetric around $x$ in the sense that $s(x+z)=s(x-z)$ for every $|z|$ small enough hence this notion can be used to get rid of symmetric but badly behaved parts of $f$ around $x$.