Modal dialogs block all mouse-clicks

Starting a few days ago, mouse-clicks are ignored on many modal dialogs or the dialog does not open. For example, when I open a Printers & Scanners ->Printer dialog, I cannot click to choose the scanner.

Many file-upload dialogs in Chrome or Firefox do not open, or if they do, the mouse-click is ignored, and I must use the keyboard to tab to the file in the file selector.

Clicking to "print" to a PDF from the printing dialog in Firefox causes the GUI to freeze.

What could be causing this? How can I diagnose it?

BigSur 11.1.6 on Macbook Pro 2017

Many modal panels are presented on behalf of a host app by system daemons and frameworks. There is an interprocess communication (IPC) dance that takes place whenever an app presents such a panel. A bug in either side of this can cause an app to appear hung on a panel.

Restarting the Mac will generally resolve problems like this. While it's impossible to know the exact cause in your case, it is likely a result of a bug affecting one of the UI frameworks or daemons, such as SystemUIServer. It's notoriously tricky to avoid deadlocks and livelocks when writing code that requires IPC, so bugs like this aren't all that surprising. (Especially when using poorly-written programs like Chrome and Firefox, which are well-known to employ a lot of nonstandard design patterns in lieu of standard APIs and thus cause all sorts of trouble beyond system instability.)

In general, to debug issues like this in the future in hopes of root-causing them, you should first collect a spindump during a period of time when you're experiencing an app hang. To do this:

1. Run the following Terminal command to enable kernel symbolication:

    sudo nvram boot-args="keepsyms=1"
   

   Make sure to copy-paste this so that you're entering straight quotes rather than curly/“smart” quotes. (If you get a permission error, you will first need to disable System Integrity Protection. You can re-enable it after you're done with this investigation.)

2. Reboot your machine and don’t open any apps except for Terminal.

3. In Terminal, enter the following command, press Return, type in your password, but do not press Return again yet:

    sudo spindump -reveal -noProcessingWhileSampling
   

4. Perform an operation that you know will be abnormally slow.

5. As soon as you begin to feel the slowness, go to the Terminal window where you entered your password and press Return so that the command begins to execute. Do not do anything else on the computer while it's running, otherwise you will distort the data collection.

6. After waiting for >10 seconds to collect a sample and another minute or two to symbolicate and format, you’ll get a file in /tmp/spindump.txt that contains a stackshot of every process.

7. Upload the file to PasteBin or some equivalent place and add the link to it to your original question. We can take a look at it and come up with next steps for the investigation.