What kind of transformation an upper triangular matrix represents

Solution 1:

Remember that the columns of a matrix are the images of a basis under the linear map that the matrix represents. The simplest observation for a triangular matrix is that the image of the $n$-h basis vector is in the span of the first $n$ basis vectors. So, the first vector gets mapped to somewhere on the line it generates, the second vector gets mapped into the plane generated by the first two vectors, and so on.

Solution 2:

For complex matrices, the Schur Theorem tells you that any matrix is unitarily equivalent to an upper triangular matrix. So, in a sense, all matrices are upper triangular.

Similarly, in the complex case, any nilpotent matrix can be represented by a strictly upper triangular matrix. So the strictly upper triangular matrices represent nilpotent matrices.