Is there a fundamental problem with extending matrix concepts to tensors?

We are familiar with the theory of matrices, more specifically their eigen-theorems and associated decompositions. Indeed singular value decomposition generalizes the spectral theorem for arbitrary matrices, not just square ones. Now it only seems natural to extend this idea of 2 dimensional array of numbers to higher dimensions, i.e. tensors. But as soon as we do this, everything breaks down.

For example even the notion of rank of matrix (which we all agree to be minimum of either column rank or row rank for a matrix) seems to be conflated when it comes to tensors. The Wikipedia page seems to use degree, order and rank of a tensor synonymously (understandably due to different terminology used in different fields, but somewhat annoying nevertheless).

"The order (also degree or rank) of a tensor is the dimensionality of the array needed to represent it, or equivalently, the number of indices needed to label a component of that array."

Also for example, the very familiar concept of eigenvalues or vectors also flies out the window (though people have defined them for super-symmetric tensors). So my question is this:

What is the fundamental reason why the "nice" theorems we have in the matrix case do not extend to the case of tensors?

I can think of a couple:

• Tensors exhibit the phenomenon of rank jumping as well as field dependence; which would imply the usual rules of analysis need to be re-examined when dealing with them.
• A large class of matrices are groups, so tools from abstract algebra are available to deal with them.
• Matrices can be viewed as operators from one space to another unambiguously, whereas viewing a tensor as an operator between spaces can get confusing very quickly.

I know there are extensions to SVD for tensors, for example Tucker decomposition, HOSVD, etc; so I am not claiming it can't be done. I also understand (somewhat) that mathematicians prefer to study tensors abstractly or using differential geometry and forms. I am just curious as to why the results generalizing the concepts form the matrix case are many; what is the underlying cause for a lack of unifying framework. The above reasons seem valid roadblocks, but do they hint at something more fundamental?

Solution 1:

I'd argue that the main point is that a matrix is (among other things) a linear map while a tensor is not.

All matrix decompositions can be seen as choosing particular bases in the domain and codomain of the linear map:

• For example LU decomposition can be written as $$ A = LU = \tilde L \begin{pmatrix}I_r & 0\\0 & 0\end{pmatrix}\tilde U $$ where $r$ is the rank and $\tilde L$ and $\tilde U$ are slight modifaction of the original LU decomposition such that they become invertible, i.e. $\tilde U$ comes from a basis in the domain and $\tilde L$ from a basis in the codomain.
• SVD is inherently a choice of bases $A = U\Sigma V^T$ with orthonormal $U$ und $V$…
• Same for spectral decomposition of symmetric $A$ (a special case of the SVD).

Ok, a matrix is also a bilinear form and a tensor is a multilinear form and, actually, this similarity can be used to define eigenvectors of tensors, but, I'd agrue that the point of view that a matrix represents a linear map is more fundamental. There a several matrices $A$ that represent the same bilinear form (e.g. $A$, $A^T$ and $(A+A^T)/2$ all represent the same bilinear form), while the linear map is unique (up to the choice of bases).

I have the impression that these notions for matrices that rely on the view as bilinear forms generalize much easier to tensors (multilinear maps) as the notions that rely on the view as a linear map.

Solution 2:

Having in mind that for a vector space $V$ there is another vectorspace $$V^*=\{V\to\Bbb F:\mbox{which are linear}\}$$ dubbed the dual then tensors are:

1) covariant tensors which are multilinear maps $$V\times...\times V\to\Bbb F,$$ 2) contravariant tensor, multilinear maps $$V^*\times...\times V^*\to\Bbb F$$ and 3) mixed tensors as multilinear maps $$V\times...\times V\times V^*\times...\times V^*\to\Bbb F.$$

In each stage one is obliged to use not only bi-indexed quantities as $$A_{ij},\ B_i{}^j,\ C^i{}_j,\ D^{ij}$$ identified as the components of a tensor and which correspond to entries of matrices. Each of those for the four possible types $$V\times V\to\Bbb F,$$ $$V\times V^*\to\Bbb F,$$ $$V^*\times V\to\Bbb F,$$ $$V^*\times V^*\to\Bbb F,$$ of bilinear maps.

But there are also tri-indexed components as $$T^{ijk}, U^{ij}{}_k,...etc$$ where you begin to have a generalization of matrices.