How to refer to "earlier" letters in the alphabet?

If I understand your question correctly, you are referring to lexicographic(al) ordering. In more general terms, A is lexicographically smaller than B if A precedes B in some alphabet X.

I don't think many people without math/computer science backgrounds would understand these terms though.

If I understand your question correctly (and to quote the words you use):

You are "actually looking for the most proper way to write the sentence [you] used as an example", namely:

Names starting with "earlier" letters come first in lists.

Personally, I would rewrite that as:

Names are listed alphabetically.
Names are listed in alphabetic order.

If you want to retain a similar format to your example:

Names starting with letters earlier in the alphabet ...
Names starting with alphabetically earlier letters ...
Names higher up the alphabet ...

but personally, I'm not keen of any of those alternatives.

More importantly, in referring to "Names starting with ... letters ...", you could be understood as implying that only the first letter is taken into consideration in determining the order, such that you could have a sequence such as:

Howe, Holmes, Hancock, Hill, Hughes

whereas true alphabetical order would take into account all letters in the name (as far a necessary) to give:

Hancock, Hill, Holmes, Howe, Hughes

"Precedes" is literally correct, although colloquially we usually say that A "comes ahead of" B in the alphabet. And, flipped around, we say B "comes after" A.

And by the way, although it is not exactly incorrect to say that 1 is "smaller than" 2, we generally say it is "less than" 2, not "smaller than."

You could say:

If your surnames start with any letter between H and P ....

Those whose surnames begin with any letter before H ...
Those whose surnames begin with any letter after P...

Edit:
Because some users have revised their answers that make mine look superfluous.

Using the OP's original example, I would like to offer the following solution:

Names starting with the first three/six letters of the alphabet come first in lists.

Not pithy I grant you, but certainly easily understood by all.