How to max out performance EC2 instance

Solution 1:

Your c5d.9xlarge comes with 900GB instance storage (aka ephemeral storage) - are you using that for storing and processing the files? While your instance has some dedicated EBS bandwidth the on-instance SSD storage will still be much faster. I suggest you use that for all source and temporary files and only store the results to EBS.

There are some caveats with instance storage though:

1. You have to format and mount it before it can be used. Refer to this answer for more details: Automatically mount SSD instance storage on AWS EC2 in Ubuntu 16.04

2. The contents is wiped when you stop and re-start the instance. It survives reboot but not stop/start.

Update: By default the SSD isn't mounted - you'll have to follow the steps in the linked answer above to make use of it. In the standard config after boot you'll be using the EBS which is slower than SSD.

Then you will have to make sure that you're actually using it - set the upload, working and temporary directory to that SSD mount point.

Or even better - since you've got 72 GB of RAM - create a RAM disk and and use that for temporary files. That will be even faster than SSD (if the files fit).

First of all disregard the upload speed and optimise the processing - upload the file to EBS and time the processing, then upload to SSD and time it and then to RAM disk and time the processing. See how much they differ.

Upload speed will be affected by many aspects, including the distance and latency between you and the AWS region you're uploading to. Are you using an AWS region close to you?

Re CPU performance - you've got 36 CPUs available, however each single core may be slower than the cores in your previous 12-core machine. It depends on the CPU architecture and clock speed. However if you can parallelise the video processing into 36 threads you should be better off on this instance. If you rely on a single thread you may not get the performance you want.

Hope that helps :)

Solution 2:

Is there any scope to look at alternatives? For the cost of a c5d.9xlarge, even if you're getting a substantial discount, in the dedicated server market you could have several equivalent or better machines

Using cloud for this kind of vertical-scaled problem is a recipe for overpaying and, as you've seen, poor performance

Sorry for the kind of non-answer, but I don't have enough rep to comment