Why do complex function have derivatives?

Solution 1:

If we define the derivative of $f(z)$ as the limit as $h$ approaches $0$ (being $h$ a complex number) of $(f(z+h)−f(z))/h$...

That's precisely what we do.

then clearly the derivative is $2z$, but what does this derivative represent??

Well, it represents what we have defined: the limit of the incremental ratio, same as in the real case. Probably you're wondering if we can interpret this complex derivative geometrically or visually, as we interpret the (real) derivative as the slope of the tangent line... There is no such a simple pictorial interpretation.

Also, shouldn't we be able to represent a complex function in 4-dimensional space, since our input and output have 2 variables each ... and then we could take directional derivatives...right?

Indeed, you could consider each of the two components (real and imaginary) separatedly, both for the variable and for the function, and then you'd get four (real) derivatives. And, yes, it's natural to ask how these four derivatives are related to the complex derivative, and if there are some (necessary and/or sufficient) restrictions on them so that the complex derivative gives the same value no matter the "direction" (as one would want)... Behold the Cauchy–Riemann equations and holomorphic functions.

Solution 2:

A complex-valued function $f$ is differentiable if near an arbitrary point $z_{0}$, the function $f$ acts like multiplication by a complex number $f'(z_{0})$ in the sense that $$ f(z_{0} + h) = f(z_{0}) + h\, f'(z_{0}) + o(|h|). $$ Since multiplication by a non-zero complex number $$ \alpha = |\alpha| \exp(i\theta) $$ rotates the plane (about the origin) through an angle $\theta$ and scales the plane (about the origin) by a factor $|\alpha|$, a complex-differentiable function is properly conformal (preserves both orientation and angle) at each point where the derivative is non-zero.

You're correct that drawing the graph of a complex-valued function of one complex variable entails four-dimensional plotting. Drawing the image of a Cartesian grid, however, is perfectly feasible in the plane:

Each map above is defined by a pair of real cubic polynomials; the red curves are the images of vertical lines; the blue curves are images of horizontal lines. The maps differ in the sign of one component of the cubic term. One map is complex differentiable (or holomorphic), one is not.

The holomorphic map sends a square grid to an "infinitesimally square grid". (There is one critical point, whose image is visually apparent.) The derivative $f'(z_{0})$ at a point determines the size of the image (quasi-)square and the angle of rotation compared to the original Cartesian square.

The non-holomorphic map sends a square grid to a grid of curvilinear parallelograms that are not generally squares. This is the geometric manifestation of the directional derivatives at $z_{0}$ depending on the direction of approach to $z_{0}$: The Cartesian grid in the domain "gets stretched and rotated by differing amounts in different directions at a single point."