Launch new instance when there is no instance open on current workspace

You need to "isolate" your current workspace. You can do that by using the Workspace Isolated Dash GNOME shell extension for example. However, since Ubuntu nowadays (version 17.10 and later) comes with Ubuntu Dock (a fork of Dash to Dock), you can activate this by simply running the following command in Terminal, no need to do anything extra:

gsettings set org.gnome.shell.extensions.dash-to-dock isolate-workspaces true

dconf-editor

in Ubuntu 20.04 LTS,
dconf-editor seems powerful (than gnome-tweaks)
and easy to use (than gsettings or dconf).

the workspace related options:

• /org/gnome/shell/extensions/dash-to-dock/isolate-workspaces
• /org/gnome/shell/app-switcher/current-workspace-only
• /org/gnome/shell/window-switcher/current-workspace-only

Run dconf-editor, press Ctrl + L to input location path.

screenshot: