profiling CUDA on nvidia 3090: "this version of nvprof doesn't support the underlying device"

I have installed CUDA toolkit on Ubuntu 20.04 LTS, on a system with nvidia rtx 3090 graphics card. and I've successfully compiled and run different code samples. But I haven't managed to run the official profiler tool, nvprof. Here is the complete error message I get:

======== Warning: This version of nvprof doesn't support the underlying device, GPU profiling skipped
======== Error: application not found.

Here is the result of nvprof --version:

nvprof: NVIDIA (R) Cuda command line profiler
Copyright (c) 2012 - 2019 NVIDIA Corporation
Release version 10.1.243 (21)

In case it is relevant, here is the result of nvcc --versioin:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Is there some compatibility issue between nvprof and the latest card from nvidia or Ubuntu 20.04?

20.04 comes with an old nvprof tool: nvidia-profiler (10.1.243-3)

20.10 comes with a newer one: nvidia-profiler (11.0.3-1ubuntu1)

Unfortunately, neither of these is capable of running on a 3000-series card.

Even when you get the 11.2 profiler from This NVIDIA server that serves deb archives, it will not support it.

Instead, you are expected to run nvidia-nsight-compute to profile your kernel. It seems like command line profiling with nvprof is no longer a thing.

Bram's answer seems to be correct. Maybe try something like this instead, if you still want to stay in the console:

nsys profile -t nvtx,cuda --stats=true --force-overwrite true --wait=all -o my_report ./my_app

nsys gives you a nice overview of the kernels and their runtimes.