Taylor expansion for vector-valued function?

Let $f:\mathbb{R}^m \to \mathbb{R}^n$. Is it possible to do a Taylor expansion of $f$ around $\theta\in\mathbb{R}^m$? I am hoping for something like

$$f\left(\theta\right) = f\left(\theta_0\right) + A \left(\theta - \theta_0\right) + \left(\theta - \theta'\right)^T \text{something} \left(\theta - \theta'\right)$$

where $A$ is a $n\times m$ matrix, and its rows are gradient of $f_i$ ($i$-the entry of vector $f$) with respect to vector $\theta$. But what should "something" be?

For example, let us consider the simple case where $f :\mathbb{R}^m \to \mathbb{R}^2$. Denote $f = \left(f_1,f_2\right)$. Then

$$f_1\left(\theta_n\right) = f_1(\theta_0) + \nabla f_1\left(\theta_0\right)(\theta_n - \theta_0) + (\theta_n - \theta_0)^T H_1(\theta') (\theta_n - \theta_0) $$

$$f_2\left(\theta_n\right) = f_2(\theta_0) + \nabla f_2\left(\theta_0\right)(\theta_n - \theta_0) + (\theta_n - \theta_0)^T H_2(\theta') (\theta_n - \theta_0) $$

If I try to put this in matrix form, I will get something like

$$f\left(\theta_0\right) = f(\theta_0) + \begin{pmatrix} \nabla f_1(\theta_0)\\ \nabla f_2(\theta_0)\end{pmatrix} (\theta_n - \theta_0) + ???$$

Solution 1:

As you seem to have worked out for yourself, you can just write the Taylor series for each component of $f$ separately; so I guess the remaining issue is how to write this down neatly.

Your "something" here is the second derivative of $f$, which is the third-order tensor comprised of the Hessian matrices $H_1,\ldots,H_n$; so you need to use a notation that can deal with this kind of object. If you're comfortable with the Einstein summation convention, then you can write $$f_i(\theta) = f_i(\theta_0) + A_{ij} (\theta_0)(\theta - \theta_0)_j + \frac 1 2 H_{ijk}(\theta')(\theta - \theta_0)_j(\theta-\theta_0)_k,$$ where $H_{ijk} = \frac{\partial^2 f_i}{\partial x_k \partial x_j}.$

Alternatively, you could go with something like $$f(\theta) = f(\theta_0) + A(\theta_0)(\theta - \theta_0) + \frac 1 2 H (\theta')(\theta-\theta_0,\theta-\theta_0)$$where $H$ is interpreted as a bilinear form on $\mathbb R^m$ taking values in $\mathbb R^n.$ You could write this using matrix multiplication as in Mostafa's answer, but this is slightly nonstandard and you should be clear that your "vector" $A$ and "matrix" $H$ are actually $\mathbb R^n$-valued.

Solution 2:

the $something$ is ,roughly say Hessian matrix. In fact according to Taylor's theorem we have:$$f\left(\theta\right) = f\left(\theta_0\right) + A \left(\theta - \theta_0\right) + \dfrac{1}{2}\left(\theta - \theta'\right)^T H(\zeta) \left(\theta - \theta'\right)$$where $\zeta$ is between $\theta$ and $\theta_0$. Also we can write$$f\left(\theta\right) = f\left(\theta_0\right) + A \left(\theta - \theta_0\right) +\dfrac{1}{2} \left(\theta - \theta'\right)^T H(\theta_0) \left(\theta - \theta'\right)+o(||\theta-\theta_0||^2)$$where $o(.)$ indicates on little-o notation.

Solution 3:

Just for the sake of completeness, Wikipedia knows such expansions under the name jet. The form of the remainder has also been discussed on this site.