Derivative of Quadratic Form

Solution 1:

Let $Q(x) = x^T A x$. Then expanding $Q(x+h)-Q(x)$ and dropping the higher order term, we get $DQ(x)(h) = x^TAh+h^TAx = x^TAh+x^TA^Th = x^T(A+A^T)h$, or more typically, $\frac{\partial Q(x)}{\partial x} = x^T(A+A^T)$.

Notice that the derivative with respect to a column vector is a row vector!

Solution 2:

You could also take the derivative of the scalar sum. \begin{equation} \begin{aligned} {\bf x^TAx} = \sum\limits_{j=1}^{n}x_j\sum\limits_{i=1}^{n}x_iA_{ji} \end{aligned} \end{equation} The derivative with respect to the $k$-th variable is then(product rule): \begin{equation} \begin{aligned} \frac{d {\bf x^TAx}}{d x_k} & = \sum\limits_{j=1}^{n}\frac{dx_j}{dx_k}\sum\limits_{i=1}^{n}x_iA_{ji} + \sum\limits_{j=1}^{n}x_k\sum\limits_{i=1}^{n} \frac{dx_i}{dx_k}A_{ji} \\ & = \sum\limits_{i=1}^{n}x_iA_{ki} + \sum\limits_{j=1}^{n}x_jA_{jk} \end{aligned} \end{equation}

If then you arrange these derivatives into a column vector, you get: \begin{equation} \begin{aligned} \begin{bmatrix} \sum\limits_{i=1}^{n}x_iA_{1i} + \sum\limits_{j=1}^{n}x_jA_{j1} \\ \sum\limits_{i=1}^{n}x_iA_{2i} + \sum\limits_{j=1}^{n}x_jA_{j2} \\ \vdots \\ \sum\limits_{i=1}^{n}x_iA_{ni} + \sum\limits_{j=1}^{n}x_jA_{jn} \\ \end{bmatrix} = {\bf Ax} + ({\bf x}^T{\bf A})^T = ({\bf A} + {\bf A}^T){\bf x} \end{aligned} \end{equation}

or if you choose to arrange them in a row, then you get: \begin{equation} \begin{aligned} \begin{bmatrix} \sum\limits_{i=1}^{n}x_iA_{1i} + \sum\limits_{j=1}^{n}x_jA_{j1} & \sum\limits_{i=1}^{n}x_iA_{2i} + \sum\limits_{j=1}^{n}x_jA_{j2} & \dots & \sum\limits_{i=1}^{n}x_iA_{ni} + \sum\limits_{j=1}^{n}x_jA_{jn} \end{bmatrix} \\ = ({\bf Ax} + ({\bf x}^T{\bf A})^T)^T = (({\bf A} + {\bf A}^T){\bf x})^T = {\bf x}^T({\bf A} + {\bf A}^T) \end{aligned} \end{equation}

Solution 3:

It is easier using index notation with Einstein (repeated sum on dummy indices) rule. That is, we can write the $i$th component of $Ax$ as $a_{ij} x_j$, and $x^T A x=x_i a_{ij} x_j = a_{ij} x_i x_j$. Then take the derivative of $f(\bf{x})$ with respect to a component $x_k$. We find \begin{eqnarray} \partial f/\partial x_k = f,_k = a_{ij} x_{i,k} x_j + a_{ij} x_i x_{j,k} = a_{ij} \delta_{ik} x_j + a_{ij} x_i \delta_{jk} = a_{kj} x_j + a_{ik} x_i, \end{eqnarray} which in matrix notation is $k$th component of ${\bf{x}}^T A + {\bf{x}}^T A^T$.

Solution 4:

Yet another approach using the Frobenius product notation.

For a column vector $x \in \mathbb{R}^n$, and a matrix $A \in \mathbb{R}^{n \times n}$ we can write:

$$ x^TAx = Tr(x^TAx) = x:Ax$$

Then we take the differential and derivative as

\begin{align} d(x:Ax) & = dx:Ax + x:Adx\\ & = Ax:dx + A^Tx:dx\\ & = (Ax + A^Tx):dx\\ \frac{\partial (x^TAx)}{\partial x} &= (Ax + A^Tx) = (A + A^T)x \end{align}