Why sometimes AWS EC2 works really slow?

I do only one thing one this server - encoding videos by ffmpeg. And sometimes it does work ok, sometimes it's really slow.

I run the same command just for a test:

$ sudo time ffmpeg -i test.mp4 -c:a copy -vf 'drawbox= : x=0 : y=0 : color=invert : t=2' output1.mp4
ffmpeg version n2.2.15 Copyright (c) 2000-2015 the FFmpeg developers
  built on Apr 30 2015 13:20:09 with gcc 4.8.2 (GCC) 20140120 (Red Hat 4.8.2-16)
......
16.24user 0.04system 0:16.30elapsed 99%CPU (0avgtext+0avgdata 60972maxresident)k
0inputs+6088outputs (0major+15811minor)pagefaults 0swaps

After 5 hours I run it again, because I saw it starts working slow again.

$ sudo time ffmpeg -i test.mp4 -c:a copy -vf 'drawbox= : x=0 : y=0 : color=invert : t=2' output1.mp4
.......
163.41user 0.84system 2:46.90elapsed 98%CPU (0avgtext+0avgdata 60988maxresident)k
0inputs+6136outputs (0major+15814minor)pagefaults 0swaps

As you can see it took 10 times more time than the first time. (1)16.24user (2)163.41user

Any ideas how can it be fixed?

Solution 1:

depending on your instance type (burstable cpu instances), you may be dealing with CPU credit.

==> https://stackoverflow.com/questions/28984106/whats-is-cpu-credit-balance-in-ec2

if you find vast fluctuations on a non-burstable instance, you may want to stop and start (not reboot) instance. this will make the instance come up on a different hypervisor (host box). it may be the current shared host has issues that will be resolved when launching vm on new host.

Solution 2:

I use a bunch of EC2 instances; I've also seen wide variance in how long the same job would take to complete - processing the same video again and again. Sometimes, two runs are very fast, variance less than 1%; other runs the variance is 5 to 10x! I've noticed that these issues occur more on our smaller instances. We have a few that are the M*.large or C*.large size and these instances do not have this issue (as much)