What is the keyboard key "Next Group"?

It's used for switching between keyboard layouts. Its keysym is ISO_Next_Group and there are similar (but I think much less used) "previous group", "first group", "last group" key codes as well.

You might recognize Shift+Alt from Windows, where it usually serves the same purpose – if you have more than one keyboard layout, of course. (It is very common e.g. in Europe to have a local layout and the US English layout, and using Shift+Alt to switch between them dates back at least to Windows 95.)

There are two uses for this key:

1. I think it got its name because the intended use is directly within a single Xkb layout, where mappings are actually done as "(keysym, group) => character" rather than just "keysym => character". As I understand it (from a quick skim), these groups are usually used for 'Shift+' and 'AltGr+' levels, but as can be seen in this Arch Wiki article, it is also possible to define keys which perform locking i.e. persistent group movements (similar to "dead keys"), allowing a single Xkb layout definition to switch between multiple languages.

2. But a more typical use is to use ISO_Next_Group as a traditional system-wide hotkey which is bound to the "Switch layouts" action in your WM/DE. Instead of being directly interpreted by Xkb, it is interpreted by XFCE's standard hotkey handler, which then instructs Xkb to activate the next layout.

3. Come to think of it, I'm not entirely sure whether any other DEs besides GNOME 3 actually handle it on their own, because Xkb comes with a built-in option grp:alt_shift_toggle that apparently enables the same layout switching within X directly. (GNOME 3 has to hook ISO_Next_Group because it actually switches between built-in Xkb layouts and IBUS-Daemon layouts, which is not applicable for most other DEs.)

   (Or maybe grp:alt_shift_toggle is what maps Alt+Shift to ISO_Next_Group?... Either way, I'm sure that I'm really mixing some things up here.)