How can I re-encode H.264 video with minimal quality loss?

I have a lot of MPEG-TS files (.TS container but H.264 video) and playback is fine except that when you skip forward/backward or fast forward it's very sluggish and gets pixelated, etc.

I've been trying to do research and I'm guessing that they were encoding with very few reference blocks (ie. it's a capture from a DVB-S satellite stream).

When I re-encode them with Handbrake (.MP4 container) they play very, very good and seeking in the video is instant, etc, etc.

Is it possible to transcode/re-encode my MPEG-TS files with minimal quality loss? If so, what is my best bet? They are each about 2 Mbps (ie. 2 GB per hour) but I don't want to re-encode them if "minimal quality loss" requires 10+ GB per file. I'm hoping to keep the video are the same size.

Can anybody give me any advice?

It's possibly a container problem, copying it to a new container format with

avconv -i input.ts -c copy output.mp4

may well fix your problem. This will be 100% lossless. If that doesn't work, a crf of 18 is normally considered 'visually lossless'; you can set this is HandBrake (under the 'video' tab), or with avconv:

avconv -i input.ts -c:a copy -c:v libx264 -crf 18 -preset veryfast output.mp4

The presets are: ultrafast, superfast, veryfast, faster, fast, medium, slower, slow, veryslow. The slower the preset, the smaller the file (but with an increase in encode time). In my personal experimentation, I've found the biggest dropoff in terms of filesize is between superfast and veryfast, after that it seems to be much more incremental.

NOTE: If you want, ffmpeg can do everything avconv can do, with identical syntax (simply replace all instances of avconv with ffmpeg).

FOOTNOTE: You may find this ffmpeg x264 encoding guide useful for more information.

Without going into the details of video encoding: There's no such thing as "minimal quality loss" when your constraint is a specific video codec (h.264) or a certain bit rate. A limiting factor will always be there – if there was a way to conserve even more quality, that'd be the holy grail of video encoding, so to speak.

There's only so much an encoder like x264 can do – your main "control" knob will be the Constant Rate Factor setting, which somehow acts as a "constant quality" factor. Apart from that, there are technical limitations on how much bitrate can be saved while re-encoding and keeping the same visual/experienced quality with one specific encoder.

Of course, you will get different results for, say, XviD (a simple MPEG 4 Part II encoder) and x264, the most advanced h.264 encoder today. And then you could also try buying the Mainconcept h.264 encoder and see if using it results in better quality at the same average bit rate.

In video encoding, you're going to have to run a few trials anyway, since it also depends on the source material. A football game will be harder to encode than a round of Snooker. There are some options in x264 that allow tuning for certain types, e.g. films, with the -tune film option in FFmpeg, for example, but the difference could be minimal.

For examples and more information see FFmpeg: The ultimate Video and Audio Manipulation Tool.

For my blu rays If I encode MPEG-TS files I use handbrake high profile and use the advanced tab with these settings: RF 20

b-adapt=2:rc-lookahead=60:bframes=8:direct=auto:trellis=2:ref=16:subq=10:me=umh:merange=24:analyse=all

It takes a while to encode but the quality is difficult to distinguished from a blu ray.

If your goal is the same size this won't happen as it's a different codec/compression. File size and quality are not the same thing.