Third order term in Taylor Series

Solution 1:

Well, $f: \mathbb{R}^n \rightarrow \mathbb{R}^n$ has multivariate Taylor expansion at $x+h$ centered at $x$ of: $$ f(x+h) = f(x)+\sum_{i=1}^n \frac{\partial f}{\partial x_i}h_i +\frac{1}{2}\sum_{i,j=1}^n \frac{\partial^2 f}{\partial x_i\partial x_j}h_ih_j+\frac{1}{6}\sum_{i,j,k=1}^n \frac{\partial^3 f}{\partial x_i\partial x_j\partial x_k}h_ih_jh_k+ \cdots$$ Here we could define $T: \mathbb{R}^n \times \mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}^n $ by its values on the basis elements $e_i,e_j,e_k$ (unit-vector basis for Cartesian coordinates in $n$-dimensions) $$ T_{ijk}=\sum_{i,j,k=1}^n \frac{\partial^3 f}{\partial x_i\partial x_j\partial x_k} $$ The question is, can we write the formula $\sum_{i,j,k=1}^nT_{ijk}h_ih_jh_k$ as a matrix multiplication of some sort... I suggest we consider $\mathbb{R}$-valued $f$ as a starting point.

(with the proper conventions set forth there is a way to write this as a matrix multiplication, but, I think in the process of doing such we lose track of the manifest trilinearity, I'm not well-versed in the needed notation at this point in time so I'll leave my answer here as it stands since expressing this as a matrix multiplication on an abstract space is not that interesting, well, at least to me today)

Solution 2:

$\renewcommand{\Re}{\mathbb{R}}$Often Taylor expansions of functions $f:\Re^n\to\Re$ at a point $x$ are meant along a given direction $d\in\Re^n$. This facilitates a lot out understanding even for first-order expansions.

Let $f\in\mathcal{C}^3$ and define a function $\phi:\Re\to\Re$ given by $\phi(\tau) = f(x+\tau d)$; this is like a slice of $f$: it describes $f$ along the line $x+\tau d,\tau\in\Re$. Then $\phi$ is three times continuously differentiable and the third order expansion of $\phi$ about $\tau=0$ is

$$\begin{aligned} \phi(\tau) = \phi(0) + \tau \phi'(0) + \tfrac{t^2}{2!}\phi''(0) + \tfrac{t^3}{3!}\phi'''(0) + o(t^3). \end{aligned}$$

But $\phi'(0)$ is related to the directional derivative of $f$ at $x$ along the direction $d$ which is

$$\begin{aligned} \phi'(0) &= \lim_{h\to 0}\frac{\phi(h) - \phi(0)}{h}\\ &= \lim_{h\to 0}\frac{f(x+h d) - f(x)}{h}\\ &= \langle \nabla f(x), d \rangle \end{aligned}$$

Let us denote this by $\nabla_{d}f(x)$.

Similary, $\phi''(0)$ can be interpreted as the directional Hessian of $f$ at $x$ along the directions $d$ and $d$, that is

$$\begin{aligned} \phi''(0) = \langle \nabla^2 f(x)d, d \rangle \end{aligned}$$

Let us denote this by $\nabla^2_{d,d}f(x)$.

The term $\phi'''(0)$ - the "Tressian" as James S. Cook jokingly put it - is more difficult to represent. Indeed, it will be a tensor as Oren explained. However, we are merely interested in the "directional Tressian" of $f$ at $x$ along directions $d$, $d$ and $d$. This construct is actually used in the context of convex optimization theory and in particular the theory of self-concordant functions and is denoted by $\nabla^3_{d,d,d}f(x)$ and we may write

$$\begin{aligned} \nabla^3_{d,d,d}f(x) = \langle \nabla^3 f(x)[d]d, d \rangle \end{aligned}$$

where $\nabla^3 f(x)$ is the third-order gradient of $f$ at $x$ which, in my opinion, is best understood via its directional variant:

$$\begin{aligned} \nabla^3f(x)[d] = \lim_{h\to 0} \frac{\nabla^2 f(x+\alpha d) - \nabla^2 f(x)}{h} \end{aligned}$$

Here $\nabla^3f(x)[d]$ is a matrix - it is a directional Hessian. Essentially, $\nabla^3f(x)[d]$ describes how the Hessian of $f$ changes at $x$ along the direction $d$.

Solution 3:

You can take a look at my paper at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2364989. I use a mapping from tensor to matrix that enables to write a fifth order Taylor series in matrix notation. Take for example the third order term. If you have a tensor $f_{xxx}$ of dimensions $n\times n\times n\times n$ whose $m,i,j,k$ element is the derivative of the $m$'th entry of $f$ wrt the $i,j,k$ entries of $x$, you can use the reshape function to reshape it into a matrix of dimensions $n\times n^3$. Then, $\frac{1}{6}g_{xxx}\left( x\otimes x\otimes x\right)$ is the third order term of the Taylor series in matrix notation.