Amazon Lambda for video and image processing?
The service I am building the architecture for has a video processing part in it. Basically, some people can upload videos (stored in S3) that will be split into frames with high resolution. From these high resolution frames, derived images (thumbnails, sprites) are built, and all of this is to be stored on S3.
My question is: where and how should I run my video processing script? I would like to use Lambda, but I'm not sure its limitations (300s timeout and 512 MB disk storage) allow me to do so. Note that the videos can weight up to several GB.
The solution I'm currently thinking about is:
- read the video from S3 using a stream (so it doesn't have to be fully copied)
- convert the video into frames, and directly upload them to S3 (I also have to make sure not to fill up the storage with the frames)
- read individually or by chunks the frames from S3, and use them to generate thumbnails and sprites, that are uploaded to S3. I'm a little bit worried here about the time that this could take, even if my Lambda is in the same region as my S3 bucket.
Do you think this is the best solution? Can you think of a simpler one? Or do you think it's a better idea to run the processing on a standard EC2 instance (maybe with some queuing system like SQS in the middle)?
Thanks!
The last thing you want is to have a valid use case (eg. large video) hit the Lambda limitations (time and/or disk storage).
You should reduce your steps into small, single-task chunks, for example:
- Convert video into frames
- Convert frames into thumbnails
- etc.
Evaluate each small task individually. Determine your min/max/average file size and calculate your min/max/average time to process. If Lambda is a suitable fit, then choose it. Otherwise, you can use EC2 instance(s) to process items from queues (eg. using SQS).
Depending on exactly what you're trying to do and whether it fits, you might find AWS' Elastic Transcoder video processing service offers a ready-to-use solution. It certainly lists thumbnail generation as part of the offering.
If you roll your own I can recommend trying Amazon SWF as I have used it to build video processing workflows before.
Like another answer suggested, you should break each step into small single-task activities and then assemble them together into a workflow. You can then run the workflows on EC2 or anywhere you like.