Is there an abstract definition of a matrix being "upper triangular"?

Another question brought this up. The only definition I have ever seen for a matrix being upper triangular is, written in component forms, "all the components below the main diagonal are zero." But of course that property is basis dependent. It is not preserved under change of basis.

Yet it doesn't seem as if it would be purely arbitrary because the product of upper triangular matrices is upper triangular, and so forth. It has closure. Is there some other sort of transformation besides a basis transformation that might be relevant here? It seems as if a set of matrices having this property should have some sort of invariants.

Is there some sort of isomorphism between the sets of upper triangular matrices in different bases?

Many true things can be said about upper-triangular matrices, obviously... :)

In my own experience, a useful more-functional (rather than notational) thing that can be said is that the subgroup of $GL_n$ consisting of upper-triangular matrices is the stabilizer of the flag (nested sequence) of subspaces consisting of the span of $e_1$, the span of $e_1$ and $e_2$, ... with standard basis vectors.

Concretely, this means the following. The matrix multiplication of a triangular matrix $A$ and $e_1$, $Ae_1$, is equal to a multiple of $e_1$, right? However, $Ae_2$ is more than a multiple of $e_2$: it can be any linear combination of $e_1$ and $e_2$. Generally, if you set $V_i= \operatorname{span}(e_1, \ldots, e_i) $, try to show that $A$ is upper triangular if and only if $A(V_i) \subseteq V_i$. The nested sequence of spaces

$$ 0 = V_0 \subset V_1 \subset \ldots \subset V_n = \mathbb{R}^n$$

is called a flag of the total space.

One proves a lemma that any maximal chain of subspaces can be mapped to that "standard" chain by an element of $GL_n$. In other words, no matter which basis you are using: being triangular is intrinsically to respect a flag with $\dim(V_i) = i$ (the last condition translate the maximality of the flag).

As Daniel Schepler aptly commented, while an ordered basis gives a maximal flag, a maximal flag does not quite specify a basis. There are more things that can be said about flags versus bases... unsurprisingly... :)

Being upper triangular is not a property of linear transformations unless you have an ordered basis. Even changing the order of a basis can change an upper triangular matrix to a matrix which is not, or vice versa.

The upper triangular $n\times n$ matrices are the second most simple form of an incidence algebra, where we take the partially ordered set to be $\{1,2,\dots,n\}$ with the usual order.

The most trivial incidence algebra is the algebra of diagonal matrices, where the order is $i\preccurlyeq j$ iff $i=j.$

One interesting thing about upper-triangular matrices is that they form an algebra even when infinite-dimensional. This is because, while multiplication requires infinite sums, all but finitely many of them are zero. You can even take upper triangular matrices with rows/columns infinite in both directions by using $(\mathbb Z,\leq)$ for your Poset.

(In a general infinite poset $P$ what is required for the algebra’s multiplication to be well-defined is for the poset to be “locally finite.”)

Here are two points regarding your question.

First, as you suggest, for any two bases the upper triangular matrices with respect to those bases are indeed related: the change of basis matrix $B$ conjugates every upper triangular matrix $M$ in one basis to an upper triangular matrix $BMB^{-1}$ in the other basis.

Second, there are indeed some useful invariants associated to upper triangular matrices, namely group theoretic invariants. This is clearest in the group $GL(n,\mathbb C)$ of all invertible $n \times n$ complex matrices. First (and this is easy) the full subgroup of upper triangular matrices is a solvable group. What's more (and this is Lie-Kolchin theorem) for every solvable subgroup $H < GL(n,\mathbb C)$ there is a basis of $\mathbb C^n$ with respect to which $H$ is upper triangular; equivalently, there exists a matrix $B \in GL(n,\mathbb C)$ such that the subgroup $BHB^{-1}$ is upper triangular.

If we consider a certain class of upper triangular matrices, namely, the ones that are strictly upper triangular, then there's indeed a nice characterization in terms of nilpotency. A matrix is nilpotent if there's a nonnegative integer $k $ for which $N^k=0$. In particular, a matrix is nilpotent if and only if it's similar to a strictly upper triangular matrix with blocks $S_1,\dots,S_n $ above the diagonal. In other words, of the form $\begin{pmatrix}0&S_1&0&\dots&0\\0&0&S_2&\dots&0\\\vdots\\0&\dots&0&0&S_n\\0&0&\dots&0&0\end {pmatrix} $.