Unable to connect to libvirt. (xubuntu)
After this is the second time this comes up I tried to reproduce it today. You already posted the answer how to install askpass (thanks), but for the other issue which was already marked a dup - here some details about it.
I installed two Ubuntu systems and set up a user ubuntu@... that can be logged in via ssh keys. One of them had libvirtd and one had virt-manager. It worked right away.
Since you mentioned askpass I removed the ability to login via keys and require a password. But in that case the error message for me seems quite clear. It directly calls for ssh-askpass in the error.
You can - as you did - install askpass, or as I outlined before set up ssh-keys - in both cases it then works fine.
Is that maybe a rather old version of virt-manager? I tried virt-manager in Xenial, Bionic, Cosmic and Disco which all worked quite similar for me.
If all of that doesn't help your case, please outline whatever is special on your user setup as it might be related.