How to bind an action if no dialog is open?

In VLC media player, I want to bind k to Space (Pause/Play) if no dialog (e.g. the preferences dialog) is open, how to achieve that?

VLC is annoying because the dialogs don't have different window classes. That means the easiest way is likely to use the window title part of WinTitle for #IfWinActive. You could also add additional criteria based on the window size itself, but that's even more annoying.

Using Window Spy from the AutoHotkey Tray Icon yields:

VLC media player
ahk_class Qt5QWindowIcon
ahk_exe vlc.exe

...or...

Simple Preferences
ahk_class Qt5QWindowIcon
ahk_exe vlc.exe

So you have generally two options in a case like this, and in some cases one will be easier than the other, depending on what you're trying to do:

1. Trigger [loosely] on the title of the main window
2. Trigger on the executable and add negative exclusion conditions for the dialogs you don't want to trigger on

So a few successive examples:

; this won't work as soon as you play a song and the title changes
#IfWinActive, VLC media player
k::Send {Space}

Next... This should mostly work, but sometimes you'll be working with a program unlike VLC where the window title changes completely, in which case you might need to resort to using only the exe...

SetTitleMatchMode, 2 ; make title match generic
#IfWinActive, VLC media player ahk_exe vlc.exe
k::Send {Space}

Next... this is too open ended, but doesn't depend on the window title at all...

#IfWinActive, ahk_exe vlc.exe
k::Send {Space}

So you can build on the case shown above and trigger on almost any window belonging to a given exe, but exclude certain dialogs...

#If WinActive("ahk_exe vlc.exe") && !WinActive("Simple Preferences") && !WinActive("whatever... keep adding conditions as needed")
k::Send {Space}

Note that for this last case using multiple conditions, we switch to using #If instead of #IfWinActive, and that you can also use #IfWinNotActive if you want to write a hotkey to be generally global with a few exceptions

The nicer/easier case is for programs that don't have all the same class for windows, in which case dialogs often have their own window class and can be included or excluded by the class part of WinTitle, vs. having only the title and exe to help differentiate what you'd like to include or exclude.