Qualitatively, what is the difference between a matrix and a tensor?

Solution 1:

Coordinate-wise, one could say that a matrix is a "square" of numbers, while a tensor is a $n$-block of numbers. But this is horrible, not insightful and even a bit wrong, since those coordinates must "change in appropriate ways" (this is part of why this is horrible).

It may be best to think as follows: given a vector space $V$, a matrix can be seen in an adequate way as a bilinear map $V^* \times V \rightarrow \mathbb{R}$ (since you asked for it, I'll not enter in details. Here, $V^*$ is the dual of $V$). A tensor can be interpreted as a multilinear map $V^* \times... \times V^* \times V \times ... \times V \rightarrow \mathbb{R}$ (not necessarily the same quantity of $V^*$'s and $V$'s).

Hence, a matrix is a kind of tensor. But tensors are more general.

Solution 2:

A rank 0 tensor is a scalar.

A rank 1 tensor is a row or column vector.

A rank 2 tensor is a matrix, often square.

A rank 3 tensor? Think 3D matrix. Instead of a rectangle with data entries for each column and row, think of a cube.

Rank 4... go 4D!