Assigning hotkeys (keyboard shortcuts) in Mac OS X 10.5.8 for audio playback

The function keys work because they are system-wide "media player control keys". Keystrokes always go to the system, which usually passes them exclusively to the active application, but some have special handling. There's a chance that you could reconfigure VLC so that the system-wide media key that means "rewind" does skip back instead. Check that out.

Otherwise you're going to need some kind of additional keyboard shortcut app like QuicKeys. (See also these suggestions). Don't give up yet! You almost certainly can cobble together a solution.

Admittedly these only work when VLC is an active menu, but they are useful. On a Mac, try this:

command + arrows (left and right) -- skip tracks

command + shift + arrows -- skip 1 minute

command + option + arrows -- skip 10 seconds

command + control + arrows -- skip 3 seconds

You can customize these shortcuts in the "preferences" panel under the main "VLC" menu (press command comma). The sixth tab is called "hotkeys." Have fun!