Have ffmpeg merge a matte key file over the normal video file removing the background

Solution 1:

Make a new mask to match the video size

Your mask is smaller than the video (568x320 vs 1280x720). Optimal solution is to make a new mask that matches the size and aspect ratio of the video. This will avoid the need to upscale the mask and will provide the best results. Avoid upscaling when possible.

If you have to use the original mask

You can use the scale2ref filter to upscale the mask to match the video:

ffmpeg -i video.mp4 -i mask.mp4 -filter_complex "[1][0]scale2ref[mask][main];[main][mask]alphamerge=shortest=1" -c:v qtrle output.mov

See the scale2ref filter documentation for more details.