How to use GPU acceleration in FFmpeg with AMD Radeon?

I'll provide instructions for both Windows and Linux, as you've not indicated what platform you're on. Use these instructions that are applicable to your case.

If you're on Windows, you can access the related encoders via AMF, aptly named h264_amf and hevc_amf, whose usage can be viewed via:

ffmpeg -h encoder=h264_amf

ffmpeg -h encoder=hevc_amf

You'd need to build an FFmpeg binary for Windows, either natively, or as recommended, cross-compiling. This project is a great start on that subject.

If you use the cross-compile option, pass the arguments below for a build with the features you'll need:

./cross_compile_ffmpeg.sh --gcc-cpu-count=12  --build-libmxf=n --disable-nonfree=n --prefer-stable=y --compiler-flavors=multi

Note that the resulting build above is non-redistributable, and is only for personal use.

On Linux, using the opensource mesa amdgpu driver (not the proprietary package installed from AMD), you can access VAAPI-based encoders, aptly named h264_vaapi and hevc_vaapi, whose usage can be viewed via:

ffmpeg -h encoder=h264_vaapi

ffmpeg -h encoder=hevc_vaapi

To the best of my knowledge (correct me if I'm wrong), VP8 and VP9 encoding isn't exposed on AMD's side via VAAPI. You can confirm this by running:

vainfo | grep Slice

On your system and confirming what encoders are supported. It's most likely that FFmpeg with VAAPI enabled is the default on the likes of Ubuntu. However, you can also build a copy from source, if so needed. See the compilation guide on the FFmpeg Wiki, and adapt as needed. You may also find helpers, such as this, useful.

There's a possibility that the AMD AMF-based encoders will eventually land on Linux, using a Vulkan-based interop. Take this as a work in progress.

On MacOS Catalina via an external GPU over Thunderbolt 3, I had success using flag

-hwaccel videotoolbox

via the default brew installed copy. I could then use default libx265 encoder, at about 20x speed over encoder via CPU.

full:

ffmpeg 
  -hwaccel videotoolbox -i 'Forget to fly.mkv' \
  -c:v libx265 -preset medium -crf 28 \
  -c:a copy \
  'Forget to fly.x265.mkv'