What's the minimal structure needed to define a notion of derivative?
Solution 1:
There are lots of ways to generalize the derivative, not all of which are compatible. This is a common feature of generalizations: for example, there are also lots of ways to generalize numbers, or infinity, or exponentiation, not all of which are compatible. Here are some generalizations of derivatives you can write down:
The Fréchet derivative of a map of Banach spaces. This is used in the theory of Banach manifolds.
A derivation on an algebra. This abstracts linearity and the Leibniz rule and can be used to define tangent vectors and vector fields in a very general setting; in particular it can be used to describe what it means for a Lie algebra to act on an algebra.
Mapping out of infinitesimal objects in suitable categories gives many notions of the derivative of a map between objects. For example, in the category of schemes over a field $k$ the "walking tangent vector" $\text{Spec } k[\varepsilon]/\varepsilon^2$ has the property that mapping out of it corresponds to taking the derivative of a map of schemes in the sense that it describes the induced map on all Zariski tangent spaces.
There is a combinatorial notion of the derivative of a combinatorial species that, upon taking exponential generating functions, recovers the derivative of a power series. It is a combinatorially meaningful operation and requires no calculus to describe.
Incidentally, integration also has generalizations like this. For example, there is a completely algebraic approach to integration involving writing down a linear functional on an algebra satisfying some conditions. It has the very desirable property of naturally including "noncommutative probability" as a special case, where there is no underlying measure space but there is still a useful notion of a (noncommutative) algebra of random variables and of expectation values of these. See this blog post for some details. This can be used to explain some aspects of quantum mechanics, and it also leads to free probability.
This is even true of limits. Some notions of convergence, such as the notion of almost-everywhere convergence of a sequence of functions on a measure space, aren't defined by a topology! See convergence space for some details.
I guess the point I'm trying to make here is that you can generalize things in all sorts of ways, depending on what you're trying to do. Don't feel burdened by the first formalism you see for doing something.