Differentiation of inner product with matrices

It is considerably simpler if you stick to vector/matrix notations.

Let $z=\mathbf{Ax}:\mathbf{Ax}$. The inner product is denoted with the colon operator here. The differential writes $$ dg = h'(z) dz - \mathbf{c}:d\mathbf{x} $$ Moreover one can show that $$ dz = 2 \mathbf{A}^T \mathbf{Ax}:d\mathbf{x} $$ So the gradient is

$$ \frac{\partial g}{\partial \mathbf{x}} = 2 h'(z) \mathbf{A}^T \mathbf{A} \mathbf{x} -\mathbf{c} $$

The crux here is how to differentiate $f(x)=\langle Ax,Ax\rangle=\|Ax\|^2$. To do this, we just expand $$f(x+\eta)=\|A(x+\eta)\|^2=\|Ax\|^2+2\langle A\eta,Ax\rangle+\|A\eta\|^2=f(x)+2\eta^\top(A^\top Ax)+o(\|\eta\|).$$ So by the definition of the derivative, we have $\nabla f=2A^\top Ax$.