Configuring Transparent Hugepages; [always], [madvise], or [never]

It's the difference between opt-in (madvise) and opt-out (always). Applications that are sensitive to THP can opt-out even under transparent_hugepages=always. For example, jemalloc-using applications are notably sensitive to it (redis, go rt, rust rt) and jemalloc offers the option at compile time as part of the allocator configuration to use MADV_NOHUGEPAGE. Starting with kernel 3.15 (RHEL7 ships with 3.10), they can opt out on a per-process basis using prctl(PR_SET_THP_DISABLE).

The default always should usually be fine because data locality is a very common property for applications making large allocations. Databases are the notable exception where they suffer for accessing small chunks across a high number of large allocations (and they can disable THP).

The kernel documentation for transparent hugepages suggests that transparent_hugepage=never is more of a debugging option than a recommended solution. It is only preferable to transparent_hugepage=madvise when the software is buggy; ie. when the developer is using MADV_HUGEPAGE incorrectly.