How can I overlay PNGs with transparency over a video? (each PNG should cover one frame)
I have a video (original.mp4), and then I have a series of PNGs with transparency that I want to overlay every frame in the video.
The strategy I'm using is:
-
Create a video from the PNGs
ffmpeg -framerate 60 -pattern_type glob -i images/*.png -c:v libx264 out.mp4
-
Normalize the two videos to have the same dimensions
ffmpeg -i out.mp4 -vf scale=1920:960,setsar=0:1,setdar=16:9 out2.mp4
ffmpeg -i original.mp4 -vf scale=1920:960,setsar=0:1,setdar=16:9 original2.mp4 -
Overlay the PNGs video and give it opacity
ffmpeg -i original2.mp4 -i heatmap2.avi -filter_complex "blend=all_mode='overlay':all_opacity=0.7" result.mp4
This almost works, but the video created from the PNGs has a black background. I've also tried encoding the video as .mov and .avi, which are supposed to retain transparency. However when I did that, the black background still came back after overlaying it.
How can I seamlessly overlay the PNGs (each is a single frame of the video) as a video with transparency over the original?
[Edit 2] From Mulvya's suggestion, here is the command I tried
ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i "images/*.png" -filter_complex "[1:v][0:v]scale2ref=iw:ih[ovr]; [ovr][0:v]blend=all_mode='overlay':all_opacity=0.7[v]" -map [v] result.mp4
And the output:
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
built with Apple LLVM version 7.0.0 (clang-700.1.76)
configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'original.mp4':
Metadata:
major_brand : mp42
minor_version : 1
compatible_brands: mp41mp42isom
creation_time : 2016-03-08 18:24:07
Duration: 00:00:06.91, start: 0.023220, bitrate: 2466 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x960 [SAR 2769:2768 DAR 2769:1384], 2331 kb/s, 29.95 fps, 30 tbr, 15360 tbn, 60 tbc (default)
Metadata:
creation_time : 2016-03-08 18:24:07
handler_name : Core Media Video
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 102 kb/s (default)
Metadata:
creation_time : 2016-03-08 18:24:07
handler_name : Core Media Audio
Input #1, image2, from 'images/*.png':
Duration: 00:00:07.10, start: 0.000000, bitrate: N/A
Stream #1:0: Video: png, rgba(pc), 1920x960, 60 fps, 60 tbr, 60 tbn, 60 tbc
File 'result.mp4' already exists. Overwrite ? [y/N] y
[libx264 @ 0x7fa5b2000c00] using SAR=2769/2768
[libx264 @ 0x7fa5b2000c00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x7fa5b2000c00] profile High, level 4.2
[libx264 @ 0x7fa5b2000c00] 264 - core 148 r2601 a0cd7d3 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[libx264 @ 0x7fa5b2001e00] using SAR=2769/2768
[libx264 @ 0x7fa5b2001e00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x7fa5b2001e00] profile High, level 4.2
[libx264 @ 0x7fa5b2001e00] 264 - core 148 r2601 a0cd7d3 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'result.mp4':
Metadata:
major_brand : mp42
minor_version : 1
compatible_brands: mp41mp42isom
encoder : Lavf56.40.101
Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1920x960 [SAR 2769:2768 DAR 2769:1384], q=-1--1, 60 fps, 15360 tbn, 60 tbc
Metadata:
encoder : Lavc56.60.100 libx264
Stream #0:1: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1920x960 [SAR 2769:2768 DAR 2769:1384], q=-1--1, 60 fps, 15360 tbn, 60 tbc
Metadata:
encoder : Lavc56.60.100 libx264
Stream mapping:
Stream #0:0 (h264) -> scale2ref:ref
Stream #0:0 (h264) -> blend:bottom
Stream #1:0 (png) -> scale2ref:default
scale2ref:ref -> Stream #0:0 (libx264)
blend -> Stream #0:1 (libx264)
Press [q] to stop, [?] for help
[image2 @ 0x7fa5b1807000] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
frame= 419 fps= 31 q=-1.0 Lq=-1.0 size= 4995kB time=00:00:07.06 bitrate=5790.8kbits/s dup=210 drop=0
video:4981kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.291158%
[libx264 @ 0x7fa5b2000c00] frame I:2 Avg QP:21.72 size:142050
[libx264 @ 0x7fa5b2000c00] frame P:118 Avg QP:23.59 size: 15323
[libx264 @ 0x7fa5b2000c00] frame B:299 Avg QP:25.88 size: 1203
[libx264 @ 0x7fa5b2000c00] consecutive B-frames: 4.8% 0.0% 0.7% 94.5%
[libx264 @ 0x7fa5b2000c00] mb I I16..4: 13.2% 60.6% 26.1%
[libx264 @ 0x7fa5b2000c00] mb P I16..4: 0.7% 0.9% 0.2% P16..4: 29.9% 7.9% 4.7% 0.0% 0.0% skip:55.6%
[libx264 @ 0x7fa5b2000c00] mb B I16..4: 0.0% 0.0% 0.0% B16..8: 16.0% 0.3% 0.1% direct: 0.1% skip:83.4% L0:35.5% L1:63.0% BI: 1.5%
[libx264 @ 0x7fa5b2000c00] 8x8 transform intra:55.6% inter:68.7%
[libx264 @ 0x7fa5b2000c00] coded y,uvDC,uvAC intra: 50.1% 59.2% 20.9% inter: 3.3% 2.8% 0.0%
[libx264 @ 0x7fa5b2000c00] i16 v,h,dc,p: 11% 47% 9% 33%
[libx264 @ 0x7fa5b2000c00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 27% 20% 4% 6% 6% 9% 4% 9%
[libx264 @ 0x7fa5b2000c00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 25% 9% 6% 8% 7% 11% 6% 9%
[libx264 @ 0x7fa5b2000c00] i8c dc,h,v,p: 50% 29% 15% 6%
[libx264 @ 0x7fa5b2000c00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fa5b2000c00] ref P L0: 65.4% 15.1% 13.9% 5.5%
[libx264 @ 0x7fa5b2000c00] ref B L0: 88.0% 8.6% 3.4%
[libx264 @ 0x7fa5b2000c00] ref B L1: 96.4% 3.6%
[libx264 @ 0x7fa5b2000c00] kb/s:2808.83
[libx264 @ 0x7fa5b2001e00] frame I:2 Avg QP:15.35 size: 38416
[libx264 @ 0x7fa5b2001e00] frame P:108 Avg QP:20.76 size: 13229
[libx264 @ 0x7fa5b2001e00] frame B:316 Avg QP:25.94 size: 3612
[libx264 @ 0x7fa5b2001e00] consecutive B-frames: 0.9% 0.0% 1.4% 97.7%
[libx264 @ 0x7fa5b2001e00] mb I I16..4: 33.8% 57.9% 8.2%
[libx264 @ 0x7fa5b2001e00] mb P I16..4: 1.0% 4.2% 0.8% P16..4: 19.9% 3.8% 2.7% 0.0% 0.0% skip:67.6%
[libx264 @ 0x7fa5b2001e00] mb B I16..4: 0.0% 0.2% 0.1% B16..8: 11.4% 1.3% 0.3% direct: 0.9% skip:85.8% L0:52.0% L1:42.9% BI: 5.1%
[libx264 @ 0x7fa5b2001e00] 8x8 transform intra:67.0% inter:77.7%
[libx264 @ 0x7fa5b2001e00] coded y,uvDC,uvAC intra: 38.2% 68.3% 42.4% inter: 2.3% 6.1% 1.9%
[libx264 @ 0x7fa5b2001e00] i16 v,h,dc,p: 51% 32% 8% 9%
[libx264 @ 0x7fa5b2001e00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19% 16% 35% 4% 5% 5% 6% 4% 6%
[libx264 @ 0x7fa5b2001e00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 21% 15% 6% 7% 7% 8% 5% 5%
[libx264 @ 0x7fa5b2001e00] i8c dc,h,v,p: 42% 25% 16% 17%
[libx264 @ 0x7fa5b2001e00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fa5b2001e00] ref P L0: 71.2% 5.7% 14.7% 8.3%
[libx264 @ 0x7fa5b2001e00] ref B L0: 90.2% 7.9% 2.0%
[libx264 @ 0x7fa5b2001e00] ref B L1: 96.8% 3.2%
[libx264 @ 0x7fa5b2001e00] kb/s:2982.61
Solution 1:
H.264, which is what libx264 encodes to, does not support transparency. Neither does mpeg4, the default codec for AVI. MOV output defaults to libx264 as well.
You can do this in one command:
ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i images/*.png \
-filter_complex "[0:v]scale=1920x960,setdar=16:9[base];[1:v]scale=1920x960,setdar=16:9[ovr];\
[ovr][base]blend=all_mode='overlay':all_opacity=0.7[v]"
-map [v] result.mp4
You don't need to scale both videos.If you just want the PNGs to scale the same as the base video, you can use scale2ref:
ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i images/*.png \
-filter_complex "[1:v][0:v]scale2ref=iw:ih[ovr][base]; \
[ovr][base]blend=all_mode='overlay':all_opacity=0.7[v]"
-map [v] result.mp4
Now, you have specified a custom DAR (1920x960 is not 16:9). If that needs to be set, use
[1:v][0:v]scale2ref=iw:ih,setdar=16:9[ovr][base];
This uses a different method to apply the overlay.
ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i "images/*.png" \
-filter_complex "[1:v][0:v]scale2ref=iw:ih[ovr][base]; \
[ovr]colorchannelmixer=aa=0.7[ovrl]; [base][ovrl]overlay[v]"
-map [v] result.mp4