"Lower number" vs. "smaller number"

My gut instinct as a PhD mathematician is that −9 is lower than −8, but not smaller than −8.

There is not a technically correct term without more context being given. If you are talking about the integer numbers (including positive and negative numbers) then I wouuld prefer to say that -9 is lower than -8. If you are talking about the magnitude of the numbers, rather than their place on the integer number line, then -8 is smaller and lower than -9 (although I would prefer to say smaller). As there is not a technically correct answer, these phrases are often used interchangeably in practice. Mathematical notation, such as -9<-8 and |-8|<|-9|, would need to be used to avoid misinterpretation.

"Less than" is also a good alternative phrase. It is the words intended by the < sign.

In summary, I would say that a small number is close to 0 and, in a context where negative numbers make sense, a low number is close to minus infinity. It is best to give more context of what kind of numbers you are talking about.

You can't argue with the whole English-speaking world, and the fact of the matter is they overwhelmingly prefer smaller to lesser or lower.

I think if pressed most people would agree that -1 is a 'bigger' number than -2. But negative numbers don't exactly have real-world correlates, so we all tend to be a bit vague on that one.

Even more vague - for those who know what imaginary numbers are - is the question of whether i is 'smaller' than 2i

LATER: I have no opinion on whether either of OP's examples is more 'correct' than the other. They're both fine to me. I'm simply making the point that of the two specific usages being asked about, on average people prefer to use the former. Here's a somewhat more specific NGram for those who still want to dispute that point...

I would generally say that ‘bigger’ refers to magnitude (distance from zero) whereas ‘higher’ refers to value (distance from negative infinity), but in context anything is possible. We don't tend to use ‘big’ or ‘small’ in mathematical contexts very much. If you want to be correct, I would recommend ‘greater’ and ‘less’. In fact, the signs < and > are read ‘is less than’ and ‘is greater than’ respectively.

5 is smaller and lower than 3, but still greater.

My math teacher used to say that in this inequality 5 is smaller than 3, but 3 is less than 5.

Joking aside, only less than and greater than are uniformly understood as < and > relation respectively; the other words (e.g. smaller, lower) are often used colloquially to mean less than, whose absolute values are less than, written smaller, etc however their usage are more ambiguous and so should be avoided when writing mathematics.

EDIT:

One could argue just about anything with Google NGram:

I believe my Calculus 2 prof used such terms as "more negative", "more positive", and "closer to zero", in case that's helpful. I know that doesn't answer the initial question. To answer that question: "smaller" is ambiguous for negative numbers, as http://en.wikipedia.org/wiki/Small_number points out; and I would suggest that "lower" is the same as "less than" but not popular for this particular phrase, getting used more in such general comparisons as "lower temperature".