Most general $A \subseteq \mathbb R$ to define derivative of $f: A \to \mathbb R$?

I personally think it is a bad idea to define the derivative of a function at a point if no neighborhood of that point is contained in the domain. The whole point is that we want a linear approximation $$f(a + h) = f(a) + f'(a) h + o(|h|)$$

to hold in a neighborhood of $a$ (hence for $h$ in some open ball) and if there are no such neighborhoods you aren't really talking about the derivative in a conventional sense, but in some weirder sense.

In other words, my inclination would be to restrict to open sets.

The definition you gave is the definition of derivative I learned at the university ;-)

Here are some links to papers which also use your definition of derivative:

• http://cs.unitbv.ro/~pascu/analysis/Derivability.pdf (page 43 – second page of pdf)

• http://math.gmu.edu/~dwalnut/teach/Math315/Summer11/5.1%20Concepts%20of%20the%20Derivative.pdf

• http://www2.math.umd.edu/~lvrmr//2009-2010-F/Classes/MATH246/CALCULUS/Derivatives.pdf (page 4 for definition of limit; page 6 for definition of derivative)

As you already said different authors give different definitions of the derivation and you will also find references with other definitions. The definition you gave is the most general one.

As you mentioned: The derivative of a function $f:A\rightarrow\mathbb{R}$ can be defined for every point of $a\in A$ for which exists at least one sequence $(a_n)$ in $A\setminus\{a\}$ with $\lim_{n\rightarrow\infty} a_n = a$.

As Leonid Kovalev already said in a comment, there are important theorems in calculus (like the mean value theorem) where the differentiable function need to be defined on an interval. That may be the reason why many authors for textbooks in calculus just define the differentiability of functions with an interval as domain.

To have a concept of differentiability for functions like $\mathbb{R}\setminus\{0\}\rightarrow \frac 1x$ too, you have to extend your definition to open subsets of $\mathbb{R}$ (because $\mathbb{R}\setminus\{0\}$ is no interval and formaly two functions differs if their domain is different). Another step of generalization leads you to the definition with limit points.

To your given example: Let $g : \mathbb{Q} \rightarrow \mathbb{R} : x \rightarrow x$. For every $x,h\in\mathbb{Q}$ we have:

$$g(x+h)=g(x)+1\cdot h$$

So every $x,h\in\mathbb{Q}$ fulfills

$$g(x+h)\approx g(x)+1\cdot h+ O(|h|^2)$$

and therefore for every $x\in\mathbb{Q}$ the derivative of $g$ at $x$ is $1$ (as you can see from the above approximiation). There is no reason why we should worry that $g$ is not defined on all $\mathbb{R}$. The approximation $g(x+h)\approx g(x)+1\cdot h+ O(|h|^2)$ is well defined (and you can think of cases where it is useful too).

Now we extend $g$ to your function $f(x) = \begin{cases} x, & x \in \mathbb Q, \\ 0, & x \notin \mathbb Q, \end{cases}$. Because there are new arguments we have to worry about, the approximation fails for every $x\in\mathbb{R}$ and therefore the extended function is not differentiable.

You have this kind of “problem” also if you just define differentiability for functions on open sets in $\mathbb{R}$. $f(x) = \begin{cases} \frac 1x & ;x\neq 0 \\ 0 & ; x=0\end{cases}$ is not differentiable everywhere but $g:\mathbb{R}\setminus\{0\}:x\mapsto \frac 1x$ is differentiable everywhere.

Or a better example from complex analysis: $f:\mathbb{R}\rightarrow\mathbb{R}:x\mapsto 1$ is differentiable but $g: \mathbb{C}\rightarrow\mathbb{R}:a+b\cdot i\mapsto\begin{cases} 1&; b=0 \\0&;b\neq 0 \end{cases}$ is not differentiable for all real numbers.

If the domain $D$ of $f$ is open in $\mathbb{R}$ you cannot change the differentiability of $f$ at points in $D$ by extending $f$ in $\mathbb{R}$. But as the above example shows you can change the differentiability of $f$ at those points by extending it to a complex function...

It is fairly common in real analysis literature (and even in some textbooks) to define the derivative of a function $f:A \rightarrow {\mathbb R},$ where $A \subseteq {\mathbb R},$ at all points in $A$ that are also limit points of $A$ (but at no other points). The earliest reasonably general treatment of this notion I know of is

Earle Raymond Hedrick, On derivatives over assemblages, Transactions of the American Mathematical Society 8 #3 (July 1907), 345-353. JFM 38.0419.02

This more general notion of a derivative does appear in certain branches of real analysis. For instance, an important technical notion is that of an approximately differentiable function. This notion is defined as the approximate limit of the usual differential quotient and as such is defined at any density point of the domain.

Approximately differentiable functions arise especially in the theory of nonabsolute integration (i.e., integrals of Denjoy-Khinchin-Perron-Kurzweil-Henstock). Russell Gordon's book is a very nice introduction to this circle of ideas.