How do I reduce RAM usage on my server?

Well, an easy solution would be to queue the Ffmpeg tasks, so only a fixed number are running at anyone time. And you should really consider running the Ffmpeg processes on a separate machine from the webserver.

This is a common structuring problem, not so much a memory problem. Sounds like you are cramming everything onto one box? DB, Web and MPG processing? This won't scale very well!

Regardless of your application, anything processing intensive will work better across multiple machines using a batch system. By spreading the load across multiple boxes, and keeping the really intensive work away from the web tier, your users will thank you!

Your web tier should only be serving the interface. You should have 1+ machines dedicated to processing video in the background. This should then become available for serving by the web tier once ready.

The best reference on this topic I have found is Building Scalable Websites by Cal Henderson, ex-CTO of Flickr. The previous link is to Amazon, so you can preview the book on the cheap. This linke to Google Books will also let you read up.

Good luck!

I think you could probably do some things to improve the memory usage, but when it is all said and done you will most likely have come out better by buying some more memory. I'm sure I will get voted down for this answer, but I'm just thinking about the economics of fixing this problem.