Can a GPU be used to Encode h264?

Using Handbrake the answer would be no. Handbrake is a CPU-only h.264 encoder, though it now is able to use OpenCL for video scaling and has some support for DXVA for video decoding.

The question linked by slhck mentions MediaCoder, but it requires either an Nvidia graphics card or an Intel CPU with a 2nd generation graphics core.

There are a couple of older products that have been obsoleted, specifically Badaboom and the ATi Avivo software.

Badaboom which could use Nvidia and Intel graphics processors to speed up the encoding process but it is now end-of-life and does not support newer graphics cards.

ATi had their own video encoder that used to be called "Avivo" before being merged into the AMD Codec Package which included the AMD Video Converter though it is no longer available.

x264 now supports OpenCL encoding acceleration. How much acceleration you get will depend on your graphics card's performance, on my GTS 450 I get none and with slower cards it actually slows down encoding. Better cards may help. All you need to do is add --opencl to the x264 command line. I believe handbrake makes use of the same x264 libraries and does offer the same OpenCL acceleration in newer builds.

https://trac.ffmpeg.org/wiki/HWAccelIntro

FFmpeg provides a subsystem for hardware acceleration.

Hardware acceleration allows to use specific devices (usually graphical card > or other specific devices) to perform multimedia processing. This allows to use dedicated hardware to perform demanding computation while freeing the CPU from such computations. Typically hardware acceleration enables specific hardware devices (usually the GPU) to perform operations related to decoding and encoding video streams, or filtering video.

When using FFmpeg the tool, HW-assisted decoding is enabled using through the -hwaccel option, which enables a specific decoder. Each decoder may have specific limitations (for example an H.264 decoder may only support baseline profile). HW-assisted encoding is enabled through the use of a specific encoder (for example nvenc_h264). Filtering HW-assisted processing is only supported in a few filters, and in that case you enable the OpenCL code through a filter option.

There are several hardware acceleration standards API, some of which are supported to some extent by FFmpeg.

"-hwaccel auto" option allows you to select the hardware-based accelerated decoding to use for the encode session. You can add this param with "auto" before input (if your x264 is compiled with OpenCL support you can try to add -x264opts param), for example:

ffmpeg -hwaccel auto -i input -vcodec libx264 -x264opts opencl output

Sorry, but I can't remember how to list all available options for "-hwaccel".

Also you can try this: https://wiki.archlinux.org/index.php/Hardware_video_acceleration