NGINX Serving Large mp4 Files extremely inefficiently
I am currently running nginx/1.0.15 on a Centos 6.6 OS. The server has the following specs:
- Intel(R) Atom(TM) CPU C2750 @ 2.40GHz (8 cores)
- 32GB Ram
- 5 x 6000 GB 7200 RPM (Raid 10)
The Problem
The server has a 1Gbit/s connection, however it tops out and bottlenecks after 400-500 mbit/s. Service starts to decline at roughly 100 connections.. and the speed with the server drops dramatically (despite having 50% bandwidth still available)
The NGINX server is strictly for serving static .mp4 files. Each file is typically 400-1200MB (700MB being the average)
I have tried many many configurations and just about all of them give me the same results.. I am extremely frustrated..
Server load also never passes 0.3.
Is there anything blatantly wrong or misguided in my configuration? Anything might help.
The Configurations
/etc/nginx/nginx.conf
user nginx;
worker_processes 9;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 51200;
use epoll;
}
worker_rlimit_nofile 600000;
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
#access_log /var/log/nginx/access.log main;
access_log off;
aio on;
sendfile off;
tcp_nopush off;
tcp_nodelay on;
#keepalive_timeout 0;
keepalive_timeout 65;
output_buffers 1 3m;
#gzip on;
include /etc/nginx/conf.d/*.conf;
open_file_cache max=10000 inactive=5m;
open_file_cache_valid 2m;
open_file_cache_min_uses 1;
open_file_cache_errors on;
}
/etc/nginx/conf.d/default.conf
server {
listen 80 default_server sndbuf=32k;
server_name _;
#charset koi8-r;
#access_log logs/host.access.log main;
include /etc/nginx/default.d/*.conf;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
location /Videos/ {
root /home;
gzip off;
gzip_static off;
mp4;
mp4_max_buffer_size 300m;
}
location /stats {
stub_status on;
}
error_page 404 /404.html;
location = /404.html {
root /usr/share/nginx/html;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
Solution 1:
The better start can be the set of following rules:
- disable logging and accept_mutex
- enable sendfile
- set sendfile_max_chunk
Configuration:
events {
accept_mutex off;
}
access_log off;
sendfile on;
sendfile_max_chunk 512k;
New Nginx (1.7.11 or newer) feature thread pool can be really helpful in your case:
location / {
root /home;
aio threads;
mp4;
}
On test samples it dramatically helps you to increase bandwidth from 1Gbps up to 9Gbps. Nine times! You have only 1Gbps but it makes it all utilised.
See more details: https://www.nginx.com/blog/thread-pools-boost-performance-9x/
Solution 2:
A good first place to start is with the actual .mp4 files, where there is usually massive areas of improvement.
So before getting lost in tuning NGINX or Apache, first tune your .mp4 files.
For this post cinematic is like a movie or television show where each frame change is required. In other words, trying to retranscode a movie like "The Croods" to 1 fps (frame/second) would reduce quality to unwatchable.
And non-cinematic refers to screen captures like Webinars our courseware posted to Udemy.
First, consider the audio component of the file. If the audio component is primarily speaking, then use ffmpeg to retranscode the file where you copy the video stream (no change) + convert the stereo stream to mono. For many .mp4 files (non-cinematic) roughly 1/3 of movie file size is video + 1/3 is left audio channel + 1/3 is right audio channel. Changing from stereo to mono, can considerably reduce file size.
Second, retranscode audio using FDK-AAC (https://github.com/mstorsjo/fdk-aac) which produces far smaller files than other aac encoders. Most modern versions of ffmpeg automagickally build FDK-AAC these days. Even Macports now builds this. One consideration, for FDK to do it's real magic requires a stereo track + when using FDK stereo audio compresses far smaller than mono, so if you're using FDK, stick with stereo.
Third, for audio reduce bitrate. Many times this is 48k, so in general use -ar 44100 (ffmpeg) or for spoken (low fi) consider dropping to 22050.
Forth, set the frame rate of your video as low as possible. So if you're doing a screen capture, a frame may only change once in 10-60 seconds, so you can drop frame rate using -r $fps, many times from 30-60 fps to 1-5 fps + quality remains same while file size plummets.
Many times I compress non-cinematic files where every 1G reduces to 10-20M.
Fifth, make sure the faststart mov atom is at the front of your files, so your files can be streamed rather than downloaded.
My ffmpeg fdk parameters...
-c:a libfdk_aac -profile:a aac_he_v2 -afterburner 1 -signaling explicit_sbr -vbr 5 -ac 2 -ar 44100
In fact, here's a typical complete ffmpeg command...
The mp4 script is just a wrapper around ffmpeg that does things like take a guess at which audio + video tracks are in english (for multitrack avi + mkv files) + then build the ffmpeg command. What's of interest is the actual command, which is the residue of years of experiments.
Try running your files through ffmpeg extreme compression first, then see if the file weights are so low/small, there's no Web server tuning required.
Areas of experiment: -r $fps + -v:crf + -v:preset + -ar bitrate
A bit of experimenting will give you the settings for smallest file size + acceptable quality.
Many of the odd options like +genpts + clearing SAR/DAR are there to ensure .mp4 files play on Roku units. These are good to keep, in case you every setup your own Roku Channel, which is a free way to reach 5,000,000+ households.
My ffmpeg command...
imac> mp4 --dr --noisy foo.avi
tc: diag=v:!h264:mpeg4,a:!aac:ac3 title='Foo (TC)' Foo-640x480-veryfast-crf18-max-tc.mp4
cd '/Users/david/Downloads/Casper.A.Spirited.Beginning.1997.DVDrip.iNTERNAL.XviD-BPDcarrier' nice -19 ffmpeg -fflags +genpts -i "foo.avi" -map 0:0 -c:v libx264 -crf:v 18 -preset:v veryfast -tune:v film -level:v 4.1 -profile:v high -bufsize:v 5000k -vf setdar=dar=0,setsar=sar=0 -x264opts colorprim=bt709:transfer=bt709:colormatrix=bt709:fullrange=off -r 29.97 -movflags +faststart -map 0:1 -c:a libfdk_aac -profile:a aac_he_v2 -afterburner 1 -signaling explicit_sbr -vbr 5 -ac 2 -ar 44100 -metadata title='Foo (TC)' -threads 0 -f mp4 -benchmark Foo-640x480-veryfast-crf18-max-tc.mp4.tmp mv -f Foo-640x480-veryfast-crf18-max-tc.mp4.tmp Foo-640x480-veryfast-crf18-max-tc.mp4
Solution 3:
Turning on multi_accept worked for me (the video used to stop about halfway and the visitor was not able to listen/watch the other half, very frustrating).
The only things I have set in nginx.conf under the events is this:
events {
worker_connections 768;
multi_accept on;
}
**It works today LOL....tomorrow we'll just have to see if it stills play fully