How can I overlay PNGs with transparency over a video? (each PNG should cover one frame)

I have a video (original.mp4), and then I have a series of PNGs with transparency that I want to overlay every frame in the video.

The strategy I'm using is:

  1. Create a video from the PNGs

    ffmpeg -framerate 60 -pattern_type glob -i images/*.png -c:v libx264 out.mp4

  2. Normalize the two videos to have the same dimensions

    ffmpeg -i out.mp4 -vf scale=1920:960,setsar=0:1,setdar=16:9 out2.mp4
    ffmpeg -i original.mp4 -vf scale=1920:960,setsar=0:1,setdar=16:9 original2.mp4

  3. Overlay the PNGs video and give it opacity

    ffmpeg -i original2.mp4 -i heatmap2.avi -filter_complex "blend=all_mode='overlay':all_opacity=0.7" result.mp4

This almost works, but the video created from the PNGs has a black background. I've also tried encoding the video as .mov and .avi, which are supposed to retain transparency. However when I did that, the black background still came back after overlaying it.

How can I seamlessly overlay the PNGs (each is a single frame of the video) as a video with transparency over the original?

[Edit 2] From Mulvya's suggestion, here is the command I tried

ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i "images/*.png" -filter_complex "[1:v][0:v]scale2ref=iw:ih[ovr]; [ovr][0:v]blend=all_mode='overlay':all_opacity=0.7[v]" -map [v] result.mp4

And the output:

ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
  built with Apple LLVM version 7.0.0 (clang-700.1.76)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'original.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp41mp42isom
    creation_time   : 2016-03-08 18:24:07
  Duration: 00:00:06.91, start: 0.023220, bitrate: 2466 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x960 [SAR 2769:2768 DAR 2769:1384], 2331 kb/s, 29.95 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2016-03-08 18:24:07
      handler_name    : Core Media Video
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 102 kb/s (default)
    Metadata:
      creation_time   : 2016-03-08 18:24:07
      handler_name    : Core Media Audio
Input #1, image2, from 'images/*.png':
  Duration: 00:00:07.10, start: 0.000000, bitrate: N/A
    Stream #1:0: Video: png, rgba(pc), 1920x960, 60 fps, 60 tbr, 60 tbn, 60 tbc
File 'result.mp4' already exists. Overwrite ? [y/N] y
[libx264 @ 0x7fa5b2000c00] using SAR=2769/2768
[libx264 @ 0x7fa5b2000c00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x7fa5b2000c00] profile High, level 4.2
[libx264 @ 0x7fa5b2000c00] 264 - core 148 r2601 a0cd7d3 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[libx264 @ 0x7fa5b2001e00] using SAR=2769/2768
[libx264 @ 0x7fa5b2001e00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x7fa5b2001e00] profile High, level 4.2
[libx264 @ 0x7fa5b2001e00] 264 - core 148 r2601 a0cd7d3 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'result.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp41mp42isom
    encoder         : Lavf56.40.101
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1920x960 [SAR 2769:2768 DAR 2769:1384], q=-1--1, 60 fps, 15360 tbn, 60 tbc
    Metadata:
      encoder         : Lavc56.60.100 libx264
    Stream #0:1: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1920x960 [SAR 2769:2768 DAR 2769:1384], q=-1--1, 60 fps, 15360 tbn, 60 tbc
    Metadata:
      encoder         : Lavc56.60.100 libx264
Stream mapping:
  Stream #0:0 (h264) -> scale2ref:ref
  Stream #0:0 (h264) -> blend:bottom
  Stream #1:0 (png) -> scale2ref:default
  scale2ref:ref -> Stream #0:0 (libx264)
  blend -> Stream #0:1 (libx264)
Press [q] to stop, [?] for help
[image2 @ 0x7fa5b1807000] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
frame=  419 fps= 31 q=-1.0 Lq=-1.0 size=    4995kB time=00:00:07.06 bitrate=5790.8kbits/s dup=210 drop=0
video:4981kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.291158%
[libx264 @ 0x7fa5b2000c00] frame I:2     Avg QP:21.72  size:142050
[libx264 @ 0x7fa5b2000c00] frame P:118   Avg QP:23.59  size: 15323
[libx264 @ 0x7fa5b2000c00] frame B:299   Avg QP:25.88  size:  1203
[libx264 @ 0x7fa5b2000c00] consecutive B-frames:  4.8%  0.0%  0.7% 94.5%
[libx264 @ 0x7fa5b2000c00] mb I  I16..4: 13.2% 60.6% 26.1%
[libx264 @ 0x7fa5b2000c00] mb P  I16..4:  0.7%  0.9%  0.2%  P16..4: 29.9%  7.9%  4.7%  0.0%  0.0%    skip:55.6%
[libx264 @ 0x7fa5b2000c00] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8: 16.0%  0.3%  0.1%  direct: 0.1%  skip:83.4%  L0:35.5% L1:63.0% BI: 1.5%
[libx264 @ 0x7fa5b2000c00] 8x8 transform intra:55.6% inter:68.7%
[libx264 @ 0x7fa5b2000c00] coded y,uvDC,uvAC intra: 50.1% 59.2% 20.9% inter: 3.3% 2.8% 0.0%
[libx264 @ 0x7fa5b2000c00] i16 v,h,dc,p: 11% 47%  9% 33%
[libx264 @ 0x7fa5b2000c00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 27% 20%  4%  6%  6%  9%  4%  9%
[libx264 @ 0x7fa5b2000c00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 25%  9%  6%  8%  7% 11%  6%  9%
[libx264 @ 0x7fa5b2000c00] i8c dc,h,v,p: 50% 29% 15%  6%
[libx264 @ 0x7fa5b2000c00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fa5b2000c00] ref P L0: 65.4% 15.1% 13.9%  5.5%
[libx264 @ 0x7fa5b2000c00] ref B L0: 88.0%  8.6%  3.4%
[libx264 @ 0x7fa5b2000c00] ref B L1: 96.4%  3.6%
[libx264 @ 0x7fa5b2000c00] kb/s:2808.83
[libx264 @ 0x7fa5b2001e00] frame I:2     Avg QP:15.35  size: 38416
[libx264 @ 0x7fa5b2001e00] frame P:108   Avg QP:20.76  size: 13229
[libx264 @ 0x7fa5b2001e00] frame B:316   Avg QP:25.94  size:  3612
[libx264 @ 0x7fa5b2001e00] consecutive B-frames:  0.9%  0.0%  1.4% 97.7%
[libx264 @ 0x7fa5b2001e00] mb I  I16..4: 33.8% 57.9%  8.2%
[libx264 @ 0x7fa5b2001e00] mb P  I16..4:  1.0%  4.2%  0.8%  P16..4: 19.9%  3.8%  2.7%  0.0%  0.0%    skip:67.6%
[libx264 @ 0x7fa5b2001e00] mb B  I16..4:  0.0%  0.2%  0.1%  B16..8: 11.4%  1.3%  0.3%  direct: 0.9%  skip:85.8%  L0:52.0% L1:42.9% BI: 5.1%
[libx264 @ 0x7fa5b2001e00] 8x8 transform intra:67.0% inter:77.7%
[libx264 @ 0x7fa5b2001e00] coded y,uvDC,uvAC intra: 38.2% 68.3% 42.4% inter: 2.3% 6.1% 1.9%
[libx264 @ 0x7fa5b2001e00] i16 v,h,dc,p: 51% 32%  8%  9%
[libx264 @ 0x7fa5b2001e00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19% 16% 35%  4%  5%  5%  6%  4%  6%
[libx264 @ 0x7fa5b2001e00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 21% 15%  6%  7%  7%  8%  5%  5%
[libx264 @ 0x7fa5b2001e00] i8c dc,h,v,p: 42% 25% 16% 17%
[libx264 @ 0x7fa5b2001e00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fa5b2001e00] ref P L0: 71.2%  5.7% 14.7%  8.3%
[libx264 @ 0x7fa5b2001e00] ref B L0: 90.2%  7.9%  2.0%
[libx264 @ 0x7fa5b2001e00] ref B L1: 96.8%  3.2%
[libx264 @ 0x7fa5b2001e00] kb/s:2982.61

Solution 1:

H.264, which is what libx264 encodes to, does not support transparency. Neither does mpeg4, the default codec for AVI. MOV output defaults to libx264 as well.

You can do this in one command:

ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i images/*.png \
-filter_complex "[0:v]scale=1920x960,setdar=16:9[base];[1:v]scale=1920x960,setdar=16:9[ovr];\
 [ovr][base]blend=all_mode='overlay':all_opacity=0.7[v]"
-map [v] result.mp4

You don't need to scale both videos.If you just want the PNGs to scale the same as the base video, you can use scale2ref:

ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i images/*.png \
-filter_complex "[1:v][0:v]scale2ref=iw:ih[ovr][base]; \
 [ovr][base]blend=all_mode='overlay':all_opacity=0.7[v]"
-map [v] result.mp4

Now, you have specified a custom DAR (1920x960 is not 16:9). If that needs to be set, use

[1:v][0:v]scale2ref=iw:ih,setdar=16:9[ovr][base];

This uses a different method to apply the overlay.

ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i "images/*.png" \
-filter_complex "[1:v][0:v]scale2ref=iw:ih[ovr][base]; \
 [ovr]colorchannelmixer=aa=0.7[ovrl]; [base][ovrl]overlay[v]"
-map [v] result.mp4