Motivation for the Nijenhuis tensor

A partial answer, regarding the history:

As far as I know, and this is also claimed in “Nijenhuis geometry” by Bolsinov, Konyaev & Matveev (arXiv:1903.04603 [math.DG]), the Nijenhuis tensor first appeared in Albert Nijenhuis's article “$X_{n-1}$-forming sets of eigenvectors” (Indag. Math. 1951, MR43540), in connection with a different problem, with no mention of complex structures: Suppose the $(1,1)$ tensor field $J$ is diagonalizable at every point. Then the Nijenhuis tensor $N_J$ is zero iff there is a local coordinate system in which $J$ is diagonal.

Let $J$ be an almost complex structure on the smooth $2n$-dimensional manifold $M$. We can extend $J$ complex linearly to the complexified tangent bundle $TM\otimes_{\mathbb{R}}\mathbb{C}$. Then $TM\otimes_{\mathbb{R}}\mathbb{C}$ decomposes as $TM\otimes_{\mathbb{R}}\mathbb{C} = T^{1,0}M\oplus T^{0,1}M$ where $T^{1,0}M$ and $T^{0,1}M$ are the $i$ and $-i$-eigenspaces of $J$ respectively. Explicitly

\begin{align*} T^{1,0}M &= \{v - iJv \mid v \in TM\}\\ T^{0,1}M &= \{v + iJv \mid v \in TM\}. \end{align*}

Moreover, given any $w \in TM\otimes_{\mathbb{R}}\mathbb{C}$, then $w = w^{1,0} + w^{0,1}$ where $w^{1,0} = \tfrac{1}{2}(w - iJw) \in T^{1,0}M$ and $w^{0,1} = \tfrac{1}{2}(w + iJw) \in T^{0,1}M$.

If $J$ is integrable, and $(U, (z^1, \dots, z^n))$ are holomorphic coordinates, then $\left\{\frac{\partial}{\partial z^1}, \dots, \frac{\partial}{\partial z^n}\right\}$ is a local basis of sections for $T^{1,0}M|_U$ and $\left\{\frac{\partial}{\partial \bar{z}^1}, \dots, \frac{\partial}{\partial \bar{z}^n}\right\}$ is a local basis of sections for $T^{0,1}M|_U$. Extending the Lie bracket complex bilinearly, it follows that sections of $T^{1,0}M$ are closed under Lie bracket, as are sections of $T^{0,1}M$, i.e. $[\Gamma(T^{1,0}M), \Gamma(T^{1,0}M)] \subseteq \Gamma(T^{1,0}M)$ and $[\Gamma(T^{0,1}M), \Gamma(T^{0,1}M)] \subseteq \Gamma(T^{0,1}M)$. Said another way, $\Gamma(T^{1,0}M)$ and $\Gamma(T^{0,1}M)$ are Lie subalgebras of $\Gamma(TM\otimes_{\mathbb{R}}\mathbb{C})$. This provides a necessary condition for integrability.

Returning to the general case, let $X-iJX, Y-iJY$ be local sections of $T^{1,0}M$. Their Lie bracket is

$$[X-iJX, Y - iJY] = [X, Y] -i[JX, Y] -i[X, JY] - [JX, JY],$$

so the $(0, 1)$-part satisfies

\begin{align*} &\ 2[X-iJX, Y - iJY]^{0,1}\\ =&\ ([X, Y] -i[JX, Y] -i[X, JY] - [JX, JY])\\ &\ + iJ([X, Y] -i[JX, Y] -i[X, JY] - [JX, JY])\\ =&\ ([X, Y] + J[JX, Y] + J[X, JY] - [JX, JY])\\ &\ + i(J[X, Y] -[JX, Y] - [X, JY] - J[JX, JY])\\ =&\ N_J(X, Y) + iJ([X, Y] + J[JX, Y] + J[X, JY] - [JX, JY])\\ =&\ N_J(X, Y) + iJN_J(X, Y). \end{align*}

Therefore $\Gamma(T^{1,0}M)$ is closed under Lie bracket if and only if $N_J = 0$. A similar computation shows that $\Gamma(T^{0,1}M)$ is closed under Lie bracket if and only if $N_J = 0$.

In conclusion, $N_J = 0$ is a necessary condition for integrability. The Newlander-Nirenberg Theorem shows that it is also sufficient.