$d_{\nabla}^2u = F_{\nabla}\circ u$?

Let $(E,\nabla)\to X$ be a vector bundle endowed with a connection. Let $u\in\Gamma(End(E))$ an endomorphism of $E$. We denote by $d_{\nabla}$ the exterior covariant derivative, induced by $\nabla$ on $\Omega^*(X,E)$ and on $\Omega^*(X,End(E))$. I saw somewhere that $$d_{\nabla}^2u = F_{\nabla}\circ u$$ but I personally find something like $$d_{\nabla}^2u=[F_{\nabla},u].$$ What is the right formula?

I guess the formula that you have looked up applies to $\Omega^*(M,E)$ and there you should get $F_{\nabla}\circ s$ in the sense of $(\xi,\eta)\mapsto F_{\nabla}(\xi,\eta)(s)$ for $s\in\Gamma(E)$.

Now applied correctly, this exactly leads to what you have obtained. This is because you have to pass from the connection on $E$ to the induced connection on $End(E)$. Calling this $\tilde\nabla$, the definitions easily imply that for the curvature you get $F_{\tilde\nabla}(\xi,\eta)(u)=[F_{\nabla}(\xi,\eta),u]$. From above you thus conclude that $d_{\tilde\nabla}^2u=F_{\tilde\nabla}\circ u=[F_{\nabla},u]$.