How can I restart the GNOME shell on Wayland?

Turning @gravity's comment into an answer, a Gnome mailing list thread says that:

runDialog: Disable restart command on wayland

Clients can be expected to deal with the WM going away temporarily, but not the display server - so when running as wayland compositor, a restart is generally a fancy way of killing the user session, and there's little we can do about it except for preventing the user to shoot herself in the foot by throwing an error.

Under Wayland, gnome-shell does not have a graceful way to restart and leave the applications open. If you must restart gnome-shell then you are probably better off saving all of your work and using the menu to log out. If that's not working for some reason, the command killall -HUP gnome-shell will get it done from the command runner (Alt+F2) or terminal.

There is no way on wayland, see this QA:

In an Xorg session one can restart GNOME shell without losing application state as applications are running against a separate server (X). But unlike Xorg in case of a Wayland session GNOME shell is not separate from the Wayland server.

So there isn't any way to restart GNOME shell in Wayland without losing application state as the display server also goes down. It's similar to restarting X server in an Xorg session.

That is the reason why this shell restart option is disabled in Wayland (recall that usually the key sequence to kill the X server is also disabled by default in the Xorg session) and there will probably never be any non-destructive way to restart GNOME shell in Wayland.

You may see this GNOME bug report for details.

But, as you already pointed out, on Xorg it is possible by just doing ALT + F2 and then entering r in the dialogue.

EDIT DEC 2021: This solution no longer works on Gnome 3.38. shell command returns

Reloading extensions does not work correctly and is no longer supported

I found a way to load/enable manually installed shell-extensions on Wayland. This article explains how to do this,

The gnome-shell command will give you the shell version currenlty installed to make sure you download the correct extension version,

> gnome-shell --version
GNOME Shell 3.30.2

in my case I have version 3.30.2. I Downloaded a shell extension for that version and proceeded to extract it content into a new folder. The name of this folder must match the uuid string found in the metadata.json file in the root of the extension. I then proceeded to create a new folder under,

mkdir ~/.local/share/gnome-shell/extensions/<uuid-string-found-in-metadata.json-file>/

After unzip the extension archive content into this folder, I used the following command to enable the the new extension,

gnome-shell-extension-tool -e <uuid-string-found-in-metadata.json-file>

the gnome-shell-extension-tool also allows you to reload an extension (-r) and disable it too (-d).