Hessian of log-sum-exp $f(z) = \operatorname{log} \sum_{i=1}^n z_i$, find $\nabla^2f(z)$

Solution 1:

Note that you have to take the derivative with respect to $x$ if you want to check convexity in $x$. This gives some more terms form the chain rule: We have $$ \frac{\partial}{\partial x_j} f(x) = \frac{\partial}{\partial z_j} \log 1^tz \cdot \frac{\partial z_j}{\partial x_j} = \frac 1{1^tz} z_j $$ Hence, \begin{align*} \frac{\partial^2 f}{\partial x_i \partial x_j} &= \frac{\partial}{\partial z_i} \left(\frac{z_j}{1^tz}\right) \cdot \frac{\partial z_i}{\partial x_i}\\ &=\frac{\delta_{ij}1^tz - z_j}{(1^t z)^2} \cdot \exp x_i\\ &= \frac{\delta_{ij}z_i\cdot 1^tz - z_iz_j}{(1^t z)^2}\\ &= \frac{\delta_{ij}z_i}{1^tz} - \frac{z_iz_j}{(1^tz)^2}\\ &= \left(\frac 1{1^tz} \mathop{\rm diag}(z) - \frac{1}{(1^tz)^2} zz^t\right)_{i,j} \end{align*}