Differentiation of vector norms

Solution 1:

$$ \frac{\partial}{\partial \beta} \left(\|F(\beta)\|^2\right) = \frac{\partial}{\partial \beta} \left(F(\beta) \cdot F(\beta)\right) = 2 \left( \frac{\partial}{\partial \beta} F(\beta) \right) \cdot F(\beta) $$ $F(\beta) \in \mathcal{R}^D$, where $D$ is the dimension of $F(\beta)$.

Solution 2:

Let's do a directional derivative instead, eventually building up to some voodoo magic.

$$a \cdot \nabla_\beta [(y - \underline X(\beta))^2 + \beta^2] = -\underline X(a) \cdot [-2(y - \underline X(\beta))] + 2 \beta \cdot a$$

But $\underline X(a) \cdot b = \overline X(b) \cdot a$. This exchanges a linear operator with its adjoint.

We can then use this to write the result as

$$2a \cdot [\overline X(\underline X(\beta)-y) + \beta]$$

Now we can take out the $a$ to get

$$2[\overline X(\underline X(\beta)-y) + \beta]$$