Why starting any GNOME based program over SSH takes about 30 seconds?

GNOME based programs always take about 30 seconds to start over SSH. Even simplest ones, like gnome-calculator. After said programs start, they run absulutely fine. I am starting SSH session using ssh <user>@<host> -YC command. I've tried checking CPU usage during this wait period, but there is nothing out of ordinary. Network bandwidth usage doesn't seem to be excessive either (in the range of 0.2 to 2 KiB/s).

I am using my Ubuntu 19.10 machine over SSH a lot and it's quite a problem for me. I've seen this problem in earlier Ubuntu versions as well.

EDIT: I've found that this delay happens when call to g_application_run() is made. Running strace explains that delay time. Last syscall before that delay is poll([{fd=11, events=POLLIN}], 1, 25000). 25000 being timeout in millisecons. All affected programs stall at this particular syscall. File descriptor 11 is created using eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK). All this means that program is waiting for some kind of event that never happens. But what can it be and what can cause it?

I believe this is due to the DISPLAY environment variable not being propagated to a local service called "xdg-desktop-portal-gtk". If it fails to start, each gnome application will try to start it during startup, wait for it to come up, then timeout after ~25s when it fails.

You can fix this manually by doing this after logging in. First, update the environment for dbus processes:

$ dbus-update-activation-environment --systemd DBUS_SESSION_BUS_ADDRESS DISPLAY XAUTHORITY

Then start the service:

$ systemctl --user start xdg-desktop-portal-gtk

Now startup should be fast again.

If someone knows the best/most elegant way to do this automagically for every using logging in to a remote system I'd like to know about it.

30 seconds sounds an awful lot like DNS timeout.

Update the question with:

time ssh localhost echo
time ssh -YC user@remote echo
time ssh -YC localhost gnome-calculator

My guess is that remote cannot make reverse lookup of your IP-address. So try adding this in server:/etc/hosts

 i.p.n.o myclient

where i.p.n.o is your clients IP address.