How to stop nvidia-settings from installing gnome desktop?

This happens because of dependencies. See terminal output from 20.04 LTS installed from mini.iso:

$ apt-cache depends nvidia-settings
nvidia-settings
  Depends: pkg-config
    pkgconf
  Depends: screen-resolution-extra
  Depends: libvdpau1
 |Depends: libgtk-3-0
  Depends: libgtk2.0-0
  Depends: libc6
  Depends: libcairo2
  Depends: libgdk-pixbuf2.0-0
  Depends: libglib2.0-0
  Depends: libjansson4
  Depends: libpango-1.0-0
  Depends: libpangocairo-1.0-0
  Depends: libx11-6
  Depends: libxnvctrl0
  Depends: libxxf86vm1
  Conflicts: 
  Replaces: 
    nvidia-settings

apt-cache depends screen-resolution-extra
screen-resolution-extra
  Depends: 
    python3
  Depends: python3-xkit
 |Depends: gnome-shell
 |Depends: policykit-1-gnome
  Depends: 
    gnome-flashback
    gnome-shell
    lxpolkit
    lxqt-policykit
    mate-polkit
    policykit-1-gnome
    polkit-kde-agent-1
    ukui-polkit
  Depends: python3-gi

In the above output you should note that gnome-shell is not located in Recommends or Suggests sections, so is not optional.

Or use packages.ubuntu.com - check pages for nvidia-settings and its dependency named screen-resolution-extra.

With known Gentoo terminology this means that Ubuntu uses gnome or gnome-shell USE-flag for nvidia-settings package.

Possible way to get minimal set of dependent package is to install lxpolkit package instead of gnome-shell as follows:

$ sudo apt install lxpolkit nvidia-settings --no-install-recommends --no-install-suggests
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  adwaita-icon-theme dbus-user-session dconf-gsettings-backend dconf-service fontconfig fontconfig-config fonts-dejavu-core glib-networking
  glib-networking-common glib-networking-services gsettings-desktop-schemas gtk-update-icon-cache hicolor-icon-theme humanity-icon-theme
  libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data libavahi-common3 libbrotli1 libcairo-gobject2
  libcairo2 libcolord2 libcups2 libdatrie1 libdbus-glib-1-2 libdconf1 libdpkg-perl libepoxy0 libfontconfig1 libgdbm-compat4 libgdk-pixbuf2.0-0
  libgdk-pixbuf2.0-common libgraphite2-3 libgtk-3-0 libgtk-3-common libgtk2.0-0 libgtk2.0-common libharfbuzz0b libjansson4 libjbig0 libjpeg-turbo8
  libjpeg8 libjson-glib-1.0-0 libjson-glib-1.0-common liblcms2-2 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libperl5.30 libpixman-1-0
  libpolkit-agent-1-0 libproxy1v5 librest-0.7-0 librsvg2-2 librsvg2-common libsoup-gnome2.4-1 libsoup2.4-1 libthai-data libthai0 libtiff5
  libunique-1.0-0 libvdpau1 libwayland-client0 libwayland-cursor0 libwayland-egl1 libwebp6 libxcb-render0 libxcb-shm0 libxcomposite1 libxcursor1
  libxdamage1 libxfixes3 libxi6 libxinerama1 libxkbcommon0 libxnvctrl0 libxrandr2 libxrender1 libxxf86vm1 lxsession-data perl perl-modules-5.30
  pkg-config policykit-1 python3-xkit screen-resolution-extra ubuntu-mono
Suggested packages:
  colord cups-common debian-keyring gnupg | gnupg2 gcc | c-compiler binutils patch git bzr gvfs liblcms2-utils librsvg2-bin perl-doc
  libterm-readline-gnu-perl | libterm-readline-perl-perl make libb-debug-perl liblocale-codes-perl dpkg-dev
Recommended packages:
  at-spi2-core libfile-fcntllock-perl libgdk-pixbuf2.0-bin libgtk-3-bin libgail-common libgtk2.0-bin vdpau-driver-all | vdpau-driver
The following NEW packages will be installed:
  adwaita-icon-theme dbus-user-session dconf-gsettings-backend dconf-service fontconfig fontconfig-config fonts-dejavu-core glib-networking
  glib-networking-common glib-networking-services gsettings-desktop-schemas gtk-update-icon-cache hicolor-icon-theme humanity-icon-theme
  libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data libavahi-common3 libbrotli1 libcairo-gobject2
  libcairo2 libcolord2 libcups2 libdatrie1 libdbus-glib-1-2 libdconf1 libdpkg-perl libepoxy0 libfontconfig1 libgdbm-compat4 libgdk-pixbuf2.0-0
  libgdk-pixbuf2.0-common libgraphite2-3 libgtk-3-0 libgtk-3-common libgtk2.0-0 libgtk2.0-common libharfbuzz0b libjansson4 libjbig0 libjpeg-turbo8
  libjpeg8 libjson-glib-1.0-0 libjson-glib-1.0-common liblcms2-2 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libperl5.30 libpixman-1-0
  libpolkit-agent-1-0 libproxy1v5 librest-0.7-0 librsvg2-2 librsvg2-common libsoup-gnome2.4-1 libsoup2.4-1 libthai-data libthai0 libtiff5
  libunique-1.0-0 libvdpau1 libwayland-client0 libwayland-cursor0 libwayland-egl1 libwebp6 libxcb-render0 libxcb-shm0 libxcomposite1 libxcursor1
  libxdamage1 libxfixes3 libxi6 libxinerama1 libxkbcommon0 libxnvctrl0 libxrandr2 libxrender1 libxxf86vm1 lxpolkit lxsession-data nvidia-settings
  perl perl-modules-5.30 pkg-config policykit-1 python3-xkit screen-resolution-extra ubuntu-mono
0 upgraded, 92 newly installed, 0 to remove and 0 not upgraded.
Need to get 26.2 MB/26.2 MB of archives.
After this operation, 133 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
...

and optionally install minimal desktop environment by