VLC Mac play videos in filename order?

Just to provide an answer rather than comments…

VLC does not consider the actual file name in its sort methods. The only name sort you can do is by Title
This title is embedded in the file [by methods above my paygrade].
Mac VLC does not "truncate the title" as claimed in another answer… so long as the internal title matches the file name. If it doesn't, then as soon as you click inside the playlist after drag & drop, the names will change to the internal title field.
Unless the media files are specifically 'de-named' the actual file name has no bearing.

If the file name & title are the same, then of course, sorting by filename is possible.

The only way I know to change this is to change the internal Title.
See this QA on our sister site Ask Different - Batch-remove mkv titles only, no re-encode for a working method to do this on mkv files. You can either change the title to the file name or rename specifically. I don't know how to achieve this on other file types.