The diffential of commutator map in a Lie group
Solution 1:
As user99914 pointed out in the comments, the Lie bracket cannot be the derivative of any function $G\times G\to G$ because it is not a linear map $\mathfrak{g}\times \mathfrak{g}\to \mathfrak{g}$ (indeed, the Lie bracket is linear if and only if it is trivial).
Nonetheless, in the realm of algebraic groups, there is a way to obtain the Lie bracket by using the commutator map. Here is how this strange story goes:-)
In what follows, to simplify the discussion, I specialise to the case $G=GL_n$, though everything holds for a general affine group scheme over any field (not even assuming smoothness, see "Pseudo-reductive Groups" by Conrad, Gabber and Prasad, section A.7). But I will make the discussion over any base field $k$ (so $GL_n$ is considered as a group variety over the field $k$). When $k=\mathbf{R}$ or $\mathbf{C}$, you can analitify group variety to get the analytic version of $GL_n$ over $\mathbf{R}$ or $\mathbf{C}$, if you wish.
First recall that the tangent bundle of $GL_n$ (as a set) is given by all maps of $k$-schemes from $\text{Spec}(k[\epsilon])\to GL_n $ (where $\epsilon ^2=0$). Short explanation: a point of $GL_n$ is a map of $k$-scheme $\text{Spec}(k)\to GL_n$, and a point with a tangent vector at that point is a map of $k$-schemes from $\text{Spec}(k[\epsilon])\to GL_n $. Hence, because $GL_n$ is affine (and by definition of affine schemes), the tangent bundle (as a set) is given by $GL_n(k[\epsilon]) = \text{Hom}_{k-\text{algebras}}(k[GL_n], k[\epsilon])=\lbrace g+X\epsilon~\vert~g\in GL_n(k) \text{ and } X\in M_n(k)\rbrace $ (where $M_n(k)$ denotes the set of all $n\times n$ matrices with coefficients in $k$). Remark: $g+X\epsilon$ is indeed an invertible $n\times n$ matrix with coefficients in $k[\epsilon]$, because $(g+X\epsilon)(g^{-1}-g^{-1}Xg^{-1}\epsilon)=1$.
Using this point of view, $\mathfrak{gl}_n(k)$ (=the tangent space of $GL_n$ at the identity) is identified with the subspace of $GL_n(k[\epsilon])$ of elements of the form $1+X\epsilon$ (or if one wishes to be more natural, one could say that the tangent space at the identity is the kernel of $GL_n(k[\epsilon])\to GL_n(k)$ obtained by setting $\epsilon=0$).
Having explained the setting, we can give the link between the Lie bracket and the commutator map: let $k[\epsilon , \delta] = k[\epsilon]\otimes_k k[\delta] $ (where $\epsilon^2 = 0 = \delta^2$). The set $GL_n(k[\epsilon , \delta])$ gives us three possible embeddings of $\mathfrak{gl}_n(k)$, that we denote $\mathfrak{g}_1, \mathfrak{g}_2$ and $\mathfrak{g}_3$. To be really explicit, $$ \mathfrak{g}_1 = \lbrace 1+X\epsilon~\vert~X\in M_n(k)\rbrace \subset GL_n(k[\epsilon , \delta])$$ $$ \mathfrak{g}_2 = \lbrace 1+X\delta~\vert~X\in M_n(k)\rbrace \subset GL_n(k[\epsilon , \delta])$$ $$ \mathfrak{g}_3 = \lbrace 1+X\epsilon \delta~\vert~X\in M_n(k)\rbrace \subset GL_n(k[\epsilon , \delta])$$
Using those embeddings, a direct and easy computation shows that the commutator map $GL_n(k[\epsilon , \delta])\times GL_n(k[\epsilon , \delta])\to GL_n(k[\epsilon , \delta])$ induces a map $\mathfrak{g}_1\times \mathfrak{g}_2\to \mathfrak{g}_3$ which is non-other than the Lie bracket!
Let me do the computation, to show you how funnily easy this is: let $(1+X\epsilon)\in \mathfrak{g}_1$ and $(1+Y\delta)\in \mathfrak{g}_2$. Then the commutator is $(1+X\epsilon)(1+Y\delta)(1+X\epsilon)^{-1}(1+Y\delta)^{-1}$. Now, since $\epsilon^2=0=\delta^2$, $(1+X\epsilon)^{-1} = (1-X\epsilon)$ and $(1+Y\delta)^{-1} = (1-Y\delta)$. Furthermore, $$(1+X\epsilon)(1+Y\delta) = 1+X\epsilon + Y\delta + XY\epsilon \delta$$ $$ (1-X\epsilon)(1-Y\delta) = 1-X\epsilon - Y\delta + XY\epsilon \delta,$$ so that in the end, multiplying everything, one gets $ 1+(XY-YX)\epsilon \delta \in \mathfrak{g}_3$, as wanted!
Note that we had to use two independent infinitesimal directions ($\epsilon$ and $\delta$!) to get the Lie bracket. If you only use one infinitesimal variable (or equivalently if you set $\delta = \epsilon$), then you get the usual derivative of the commutator map, which is trivial. This triviality was explained in the other answer, and our little computation also shows it, since in this case $\epsilon \delta = \epsilon^2 = 0$.
Solution 2:
I think the differential at $(e,e)$ is actually zero:
First, if $m : G\times G \to G, (g,h)\mapsto gh$ is the multiplication map then $dm : T_e G \oplus T_e G \to T_e G$ is given by $(X,Y)\mapsto X+Y$.
Also if $i: G \to G, g \mapsto g^{-1}$ denotes the inversion map then we have $di : T_e G \to T_e G, X\mapsto -X$.
Your map $f$ is given by the composition $$ G\times G \to G\times G\times G \times G \to G\times G \to G \\ (g,h) \mapsto (g,h, g^{-1}, h^{-1}) \mapsto (gh, g^{-1} h^{-1}) \mapsto ghg^{-1} h^{-1} $$ so the differential is the composition $$ (X,Y) \mapsto (X,Y, -X,-Y) \mapsto (X+Y, -X-Y) \mapsto0. $$
Solution 3:
Let $G$ be a Lie group with the neutral element $e$. Then, given two elements in its Lie algebra, $\xi,\eta\in\operatorname{Lie} G:=T_eG$, represented by (smooth) paths $x,y\colon (-\varepsilon,\varepsilon)\to G$ with $x(0)=y(0)=e$, one can define the Lie bracket as the second derivative of the commutator map: $$[\xi,\eta]=\frac{d^2}{dt^2}(x(t)y(t)x(t)^{-1}y(t)^{-1})\biggr|_{t=0}.$$ Note that the first derivative of the commutator vanishes, which makes the second derivative well-defined as an element of the tangent space $T_eG$.