Why is sort -k5nr not a syntax error?

Because sort is implemented in a way that parses these in the way you expect.

See also here:

• Sometimes options and their arguments are run together, sometimes separated by whitespace, and sometimes by a character, typically : or =. Thus "Prog -fFilename", "Prog -f Filename", "Prog -f:Filename", "Prog -f=Filename".
• Some programs allow single-character options to be combined; others do not. The switch "-fA" may mean the same as "-f -A", or it may be incorrect, or it may even be a valid but different parameter.

This looks like a combination of both (works without whitespace, and combination of single-character options).

They are simply different programs whose argument parsing is implemented differently.

In coreutils 8.13, compare the following::

• src/sort.c line 4315, invoking the special integer parsing function parse_field_count, that returns with the first invalid character (i.e. once the number value is finished and the next option starts): That's why sort can handle your arguments.
• src/cut.c, line 803 ff., simply using the regular getopt behavior of interpreting everything until the next white space as parameter to the current option.