How can I use CRF encoding with nvenc in ffmpeg?

For CRF-based encodes, pass the following arguments in the snippet below to FFmpeg, assuming you're using an up-to-date build (as at the moment):

-c:v h264_nvenc -preset:v p7 -tune:v hq -rc:v vbr -cq:v 19 -b:v 0 -profile:v high

A fixed CQ value of 19 is the recommended setting as its' visually identical to 0, yet preserves good compression trade off to file size. See this write-up for more on what CRF does.

Note that the -cq scale is logarithmic, meaning that 0 is essentially lossless and 51 would be the absolute worst.

Quality can be further improved upon by adding options such as B-frames (limit this to 3, at most, and this requires the H.264 Main profile and above).

Baseline profiles do not support B-frames. To enable B-frames,pass -bf {uint} to the video encoder, such that -bf:v 4 would result in the encoder using 4 B-frames, and this can be dynamically tuned by the encoder by optionally enabling look-ahead, via the private codec option -rc-lookahead:v n, where n ranges between 32-40.

The key parts here are the -cq:v 19 and the -rc:v vbr arguments, which allow you to tune the encoder with the Constant quality mode in VBR rate control while adhering to a CRF value of 19.

And now, small notes about NVENC, and tuning it for high quality encodes:

NVENC, like any other hardware-based encoder, has several limitations, and in particular with HEVC, here are the known limitations:

1. On Pascal:

For HEVC encodes, the following limitations apply:

• CTU sizes above 32 are not supported.
• B-frames in HEVC are also not supported.
• The texture formats supported by the NVENC encoder limit the color spaces that the encoder can work with. For now, we have support for 4:2:0 (8-bit) and 4:4:4 (for 10-bit). Extraneous formats such as 4:2:2 10-bit are not supported. This will affect some workflows where such colorspaces are required.
• Look ahead control is also limited to 32 frames. You may want to look at this editorial for more details.

Turing has all the enhancements available to Pascal, with the addition of B-frame support for HEVC and the ability to use B-frames as a reference. See this answer for an example on this capability.

2. And on Maxwell Gen 2 (GM200x series GPUs):

HEVC encoding lacks the following features:

• Sample Adaptive Offset (SAO) loop filter capabilities.
• Adaptive quantization
• Look-ahead rate control.

The impact here for Maxwell is that motion heavy scenes with HEVC under constrained bitrates may suffer from artifacting (blockiness) due to the missing lookahead functions and adaptive sample offset (SAO) loop filtering capabilities. Pascal and Turing has somewhat improved on this capability, but depending on the version of the SDK that the video encoder was built with, not all features may be available.

For instance, weighted prediction mode for H.264 encodes on Pascal requires NVENC SDK 8.0x and above, and this encode mode will also disable B-frame support. Likewise, the combination of hardware-based scalers running off the Nvidia Performance Primitives (NPP) with NVENC may introduce performance improvements with video scaling applications at the cost of scaling artifacting, particularly with upscaled content. The same also impacts the video encode pipeline as NPP's scaling functions run off the CUDA cores on the GPU, and as such, the performance impact introduced by the extra load should be analyzed on a case-by case basis to determine if the performance-quality trade-off is acceptable.

Keep this in mind: A hardware-based encoder will always offer somewhat lesser customization than an equivalent software-based implementation, and as such, your mileage and acceptable output quality will always differ.

With the current NVENC encoder wrapper implementation in FFmpeg, note that valid presets can now be overriden with a tunable, depending on the target workload. See this answer on how this is implemented, as shown in the example given above.

And for your reference:

With FFmpeg, you can always refer to an encoder's settings for customization by:

ffmpeg -h encoder {encoder-name}

So, for NVENC-based encoders, you can run:

ffmpeg -h encoder=hevc_nvenc

ffmpeg -h encoder=h264_nvenc

You can also see all the NVENC-based encoders and NPP-based scalers (if built as such) by running:

for i in encoders decoders filters; do
    echo $i:; ffmpeg -hide_banner -${i} | egrep -i "npp|cuvid|nvenc|cuda"
done

Sample output on my testbed:

encoders:
 V..... h264_nvenc           NVIDIA NVENC H.264 encoder (codec h264)
 V..... nvenc                NVIDIA NVENC H.264 encoder (codec h264)
 V..... nvenc_h264           NVIDIA NVENC H.264 encoder (codec h264)
 V..... nvenc_hevc           NVIDIA NVENC hevc encoder (codec hevc)
 V..... hevc_nvenc           NVIDIA NVENC hevc encoder (codec hevc)
decoders:
 V..... h263_cuvid           Nvidia CUVID H263 decoder (codec h263)
 V..... h264_cuvid           Nvidia CUVID H264 decoder (codec h264)
 V..... hevc_cuvid           Nvidia CUVID HEVC decoder (codec hevc)
 V..... mjpeg_cuvid          Nvidia CUVID MJPEG decoder (codec mjpeg)
 V..... mpeg1_cuvid          Nvidia CUVID MPEG1VIDEO decoder (codec mpeg1video)
 V..... mpeg2_cuvid          Nvidia CUVID MPEG2VIDEO decoder (codec mpeg2video)
 V..... mpeg4_cuvid          Nvidia CUVID MPEG4 decoder (codec mpeg4)
 V..... vc1_cuvid            Nvidia CUVID VC1 decoder (codec vc1)
 V..... vp8_cuvid            Nvidia CUVID VP8 decoder (codec vp8)
 V..... vp9_cuvid            Nvidia CUVID VP9 decoder (codec vp9)
filters:
 ... hwupload_cuda     V->V       Upload a system memory frame to a CUDA device.
 ... scale_npp         V->V       NVIDIA Performance Primitives video scaling and format conversion

For -crf replacement from libx264 may be -cq or -qp from h264_nvenc:

-crf Select the quality for constant quality mode

-cq Set target quality level (0 to 51, 0 means automatic) for constant quality mode in VBR rate control

-qp Constant quantization parameter rate control method (from -1 to 51) (default -1)

Fastest hardware accelerated encode method:

ffmpeg -hwaccel cuvid -c:v h264_cuvid -resize 640x480 -i input.mp4 -c:v h264_nvenc -cq 21 -c:a copy output.mp4

-resize resolution on input (in hardware); no need for ffmpeg to be compiled with --enable-libnpp for scale_npp filter.

For more info:

ffmpeg -h encoder=h264_nvenc

ffmpeg -h denoder=h264_cuvid

I'm not an authority on this but I have done a good amount of research, specifically for getting Bluray rips archived with indistinguishable and transparent compression compared to the original rip for files with HEVC/h.265 video streams using Nvidia hardware encoder acceleration. I take care when I do this, so before committing to any particular setting, I recommend to test two particular parts of each movie. The 2 parts I choose are the darkest part of the movie and a fast-motion part.

I passthru the audio on both tests and only render about 1 minute for a couple reasons: The look-ahead needs a prescribed number of frames, and the bitrate fluctuates from scene to scene. Here is what I use with FFMPEG to test out 1 minute of the video from 30 minutes 00 seconds to 31 minutes 00 seconds.

Furthermore, I test the file with a -qp of 22, 23, 24 for HEVC/h.265. I can't tell the difference going any lower. Going higher than 24 does start to get color banding on dark scenes or on scenes with a glowing light or a sky with a smooth grandiance from white to blue.

ffmpeg -hwaccel auto -ss 00:30:01 -to 00:31:01 -i input.mkv -map 0:v:0 -map 0:a:0 -map 0:s:0 -c copy -c:v hevc_nvenc -rc constqp -qp 24 -b:v 0K -c:a copy output.mkv

The -map 0 respectively, demuxes the 1st Video, Audio, and Subtitle streams (although Subtitle is not technically a stream, I just call it that for the sake of simplicity) and assigns them to the first stream in the remuxed file. The rest of the settings are explained earlier on this page or can be looked up.

TAKE NOTE:
The container must be MKV to remux in the Subtitles...for whatever reason...I just couldn't get MP4 to work. Again, I'm no pro at this but the results I've gotten from this are exceptionally good.

After I check the 1 minute test files and like what I see, I then go ahead and process the movies without the -ss 00:30:01 -to 00:31:01. I suggest that when you look at the file, to play back the 1 minutes renders frame by frame.

I believe I found a solution:

ffmpeg -hwaccel auto -i in.mp4 -c:v h264_nvenc -preset llhq -rc constqp -qp 21 -c:a copy out.mp4

It seems that h264_nvenc uses -qp instead of -crf. This option only works while -rc is set to constqp.