Cardinal or ordinal numbers when the word "number" is used

I've revised and extended my original answer.

In common usage, a cardinal number is a number used in counting (a counting number), such as 1, 2, 3, .... Weisstein, Eric W. "Cardinal Number." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/CardinalNumber.html

… [A]n ordinal number is an adjective which describes the numerical position of an object, e.g., first, second, third, etc. (Weisstein, Eric W. "Ordinal Number." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/OrdinalNumber.html)

The number, “three", is cardinal and "third" is ordinal. Nonetheless, in the sentence, "He was number three to finish the race", ordinal numbers merit some attention.

Imagine a collection of runners at the finish line, discussing the results of the race.

One runner says, " I was number three to finish the race".

Another says, "I was number two."

Still another runner says, "I was number one"

These runners are discussing their ranks, that is, the order in which they crossed the finish line. They are not discussing how many runners are in a set.

Cardinal numbers are being employed to assign positions to runners along an ordering. This is possible because the cardinal numbers are themselves ordered.

For the case at hand, the cardinal number three is being used as a “figure of speech” (https://en.wikipedia.org/wiki/Figure_of_speech) for the ordinal number, “third”.

Cardinal.

I always think that ordinal numbers tell what order something happened in: first, second, third...

But we don't use the construction "number second* for anything. It's always "number [cardinal]."