How can I get high quality/low size MP4s like the LOL release group?

The guidelines for encoding (pirated) video are called "scene guides". They are signed by various release groups who will make sure their videos comply to the guide, LOL being one of these groups.

You can find a list of scene guides scenerules.org. The standard you're looking for is—if I'm not mistaken—the SD x264 TV Releasing Standard, assuming that your video is 1280×780. If you have HD video, you're probably not going to get really good quality at 200 MB for 45 minutes. That'd translate to roughly 600 kBit/s and would be enough for 720p though.

As for encoding tools, you can get pretty far with FFmpeg (which Handbrake is based on). The FAAC audio encoder is banned, but FFmpeg doesn't support it anymore anyway. You can use another high quality AAC encoder such as libfdk-aac.

If you have no prior experience with video encoding and all of these keywords don't mean anything to you, please have a look at the following blog articles / Super User questions:

• What is a Codec (e.g. DivX?), and how does it differ from a File Format (e.g. MPG)?
• FFmpeg: The ultimate Video and Audio Manipulation Tool

Basically, the rules say you have to use the x264 encoder with a Constant Rate Factor quality setting. What it does is quite different than setting a constant bitrate: you can set the CRF to have x264 encode the video with a constant quality.

Typical CRF levels are between 18 and 28, and the release group recommends 19–20 for high quality encoding of material that is easy to compress. You're not going to know the final file size in advance when doing that, but that's the whole point of doing CRF encoding: You don't care about the actual file size—you just want good quality.

Once you set up ffmpeg, and you have your input video ready, encoding is actually quite simple. Note however that your source video must be of high quality, otherwise re-encoding will only deteriorate its quality much more.

Here are the key parameters:

• x264 video with:
• CRF 19–20
• slow profile (better compression efficiency at the expense of encoding time)
• H.264 level 3.1, which sets a few constraints
• Downscaling has to be done with any resizer other than bicubic (which is the FFmpeg-default). Lanczos can be used, for example.
• A tune setting for the source (e.g. film for film-like content)
• AAC audio with:
• Variable bit rate encoding
• Bit rate between 96 and 160 kBit/s

In ffmpeg, this translates to the following:

ffmpeg -i input.mp4 \
-c:v libx264 -crf 19 -level 3.1 -preset slow -tune film \
-filter:v scale=-1:720 -sws_flags lanczos \
-c:a libfdk_aac -vbr 5 \
output.mp4

In FFmpeg, your choices for AAC encoding are a bit limited. See here for your AAC choices and also refer to the FFmpeg and AAC Encoding Guide for more information.

If your video ends up being too large, you may adjust the CRF. Values ±6 will result in half/double the bit rate, so if you start with 19 and go to 25, you'd end up with half the bitrate. The scene rules don't allow CRF above 24, but you probably won't mind.

Can't speak to the release group you mentioned since you didn't post the mediainfo. But here's a guide to making compact encodes, which means striking a good quality/size balance.

You should start with HandBrake and the best available source. In the Source section, select a short but difficult to encode segment (with both action and fine detail). In the Destination section, select libmkv. In the Video tab, uncheck use advanced options (you can still add those options case by case in more settings), select --rc=crf ratecontrol, the highest --preset your computer can bear (slower or veryslow), a --tune that matches the source (film or animation should cover it), and set the profile and level to auto/auto. Then play with the resolution (only if the source is HD; in picture settings, set anamorphic to loose and vary one of the axes, from the next highres resolution and down) and the --crf until you find a good compromise between quality and bitrate for your short segment. Note that the crf is tied to the rest of your encoding settings, which is why it should be the last video setting you change. In Audio, go with Vorbis at 160kbps (this will use AoTuV -q5, achieving transparency without further tuning). In the other tabs, check that everything you might need is preserved (chapter marks, subtitle tracks, metadata, etc). When you're satisfied, you can use these settings to encode the rest of the episode; the average bitrate should end up much lower than on the difficult segment. When you're confident in your settings, you can use them to encode whole seasons.

How the above steps improve compactness: libmkv gives you access to more codecs, profile and version to auto means there won't be any playback constraints to hold back the encoder. A short segment (a few seconds) gives you a lot more time to tweak things and look at the results. Tune optimises psychovisual (perceived) quality for what the source is supposed to look like. Higher presets improve compactness at the expense of encode time (this may not be noticeable on your difficult segment, but will improve the bitrate in most other scenes). CRF ratecontrol means that your encode will vary the bitrate to target a given quality, with more demanding episodes getting more space.

A denoising filter can also be very useful if you're keeping the original resolution. You'll want to keep it light because the goal isn't to clean up a perceptible source defect, but to improve compressibility without noticeably degrading quality. Here is a guide to using HandBrake's HQDN3D denoiser, which is in between x264's --nr and state of the art denoisers for quality.

FFmpeg TV

Based on slhck answer I built this simple Python solution to call the right ffmpeg parameters. Based on the latest scenerules 2016 TV standards.

python3 ffmpeg-tv.py input.mp4 output.mkv

Described you can find all the parameters, and a brief description on why they were set this way by default.

  # 1.3) Providers which downscale 1080i to 720p (e.g. BellTV) are not allowed.
  # 5.10) Resized video must be within 0.5% of the original aspect ratio.
  scale="-1:-1" # <width>:<height>; -1 maintain ratio.

  # 4.1) Video must be H.264/MPEG-4 AVC encoded with x264 8-bit.
  video_encoder="libx264"

  # 4.4) Constant Rate Factor (--crf) must be used.
  # 4.4.1) CRF values below 18 and above 23 are never allowed.
  # http://slhck.info/video/2017/02/24/crf-guide.html
  # (lossless) 0 <- (better) 23 <- (worst) 51
  # ┌─────────────────┬───────┬───────────────────────────────────────────┐
  # │ Compressibility      │  CRF    │ General Examples                                      │
  # ├─────────────────┼───────┼───────────────────────────────────────────┤
  # │ High                 │ 18-19   │ Scripted, Talk Shows, Animation, Stand-Up             │
  # │ Medium               │ 20-21   │ Documentary, Reality, Variety, Poker                  │
  # │ Low                  │ 22-23   │ Sports, Awards, Live Events                           │
  # └─────────────────┴───────┴───────────────────────────────────────────┘
  crf="19"

  # 4.6) Settings cannot go below what is specified by preset (--preset) 'slow'.
  preset="slow"

  # 4.7) Level (--level) must be '4.1'.
  level="4.1"

  # 6.4) Only sharp resizers, such as Spline36Resize, BlackmanResize or LanczosResize/Lanczos4Resize,
  # must be used.
  # 6.4.1) Simple resizers, such as Bicubic, PointResize or Simple, are not allowed. 
  resizer="lanczos"

  # 4.17) Optional tuning (--tune) parameters allowed are: 'film', 'grain' or 'animation'. 
  # https://superuser.com/questions/564402/explanation-of-x264-tune
  # film – intended for high-bitrate/high-quality movie content. Lower deblocking is used here.
  tune="film"

  # 8.1) Audio must be in the original format provided.
  # 8.1.1) Transcoding audio is not allowed.
  # 8.2) Multiple language audio tracks are allowed.
  # Since we cannot ensure that the external content is in an acceptable TV format, we'll recode
  # it into aac. We're just playing safe here.
  # 
  # FFmpeg supports two AAC-LC encoders (aac and libfdk_aac) and one HE-AAC (v1/2) encoder
  # (libfdk_aac). The license of libfdk_aac is not compatible with GPL, so the GPL does not permit
  # distribution of binaries containing incompatible code when GPL-licensed code is also included.
  # libfdk_aac is "non-free", and requires ffmpeg to be compiled manually.
  # Second best encoder is the native FFmpeg AAC encoder. (aac)
  audio_encoder="aac"

  # Audio quality (bit rate).
  # Use either VBR or CBR. VBR is the easiest.
  # https://trac.ffmpeg.org/wiki/Encode/AAC#fdk_vbr
  # VBR: Target a quality, rather than a specific bit rate. 1 is lowest quality and 5 is highest
  # quality.
  # https://trac.ffmpeg.org/wiki/Encode/AAC#fdk_cbr
  # http://wiki.hydrogenaud.io/index.php?title=Fraunhofer_FDK_AAC#Bitrate_Modes
  # CBR: kbps
  vbr="5"
  cbr=None

  # 8.2) Multiple language audio tracks are allowed.
  # 8.2.1) The default audio track must be the language intended for release (e.g. An English release
  # containing English, German and Russian audio tracks, must have the default flag set on the English
  # track).
  # https://trac.ffmpeg.org/wiki/Map
  # Include "all" inputs to the output: -map 0
  map="0"

See ffmpeg-tv.py for the complete source code.