How can I hover over icons in Ubuntu Dock to see all windows of one application?

I am not aware of any window-preview-on-mouse-hover feature, but you can activate previews of open windows on

• Primary mouse button click by running

  gsettings set org.gnome.shell.extensions.dash-to-dock click-action 'previews'
  

• shift+click by running

  gsettings set org.gnome.shell.extensions.dash-to-dock shift-click-action 'previews'
  

• Middle mouse button click by running

  gsettings set org.gnome.shell.extensions.dash-to-dock middle-click-action 'previews'
  

• shift+middle mouse button click by running

  gsettings set org.gnome.shell.extensions.dash-to-dock shift-middle-click-action 'previews'
  

(To find out all possible options for an action, run

gsettings range org.gnome.shell.extensions.dash-to-dock click-action

and so.)

As a side note, a GNOME extension called Dash to Panel puts the application icons to the top bar (and provides an option to move top bar to bottom):

It also provides the window-preview-on-mouse-hover feature:

_{(source: official GitHub page)}

For Ubuntu 20.04

gsettings set org.gnome.shell.extensions.dash-to-dock click-action minimize-or-previews