How to measure GPU usage?

  • For Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU.
  • For Intel GPU's you can use the intel-gpu-tools.
  • AMD has two options

    1. fglrx (closed source drivers):

      aticonfig --odgc --odgt
      
    2. And for mesa (open source drivers), you can use RadeonTop Install via the software center.

Source:GPU usage monitoring


You can use gpustat, which is a simple command-line script (wrapper for nvidia-smi) for querying and monitoring GPU status:

enter image description here


Nvidia: to continuously update the output of nvidia-smi, you can use nvidia-smi --loop=1 (refresh interval of 1 second) or nvidia-smi --loop-ms=1000 (refresh interval of 1000 milliseconds).

   -l SEC, --loop=SEC
       Continuously  report  query data at the specified interval, rather than
       the default of  just  once.   The  application  will  sleep  in-between
       queries.   Note  that on Linux ECC error or XID error events will print
       out during the sleep period if the -x flag was not specified.  Pressing
       Ctrl+C at any time will abort the loop, which will otherwise run indef‐
       initely.  If no argument is specified for the -l form a default  inter‐
       val of 5 seconds is used.

   -lms ms, --loop-ms=ms
       Same as -l,--loop but in milliseconds.

FYI:

  • How to log GPU load?
  • GPU utilization is N/A when using nvidia-smi