How do I go about VMs in the cloud I only need for specific times?

I am currently doing processing of videos on a local server which I'd like outsource to the Cloud. I am quite familiar with the Google Cloud which is why I would choose Google. The process would consist of:

• Upload of video files to Cloud Storage
• Processing of file in a Compute Engine instance
• Storing of processed video to Cloud Storage

In average I would need the Compute Engine instance for 1 hour a day. Is there a simple way to automatically power up the instance and shut it down on demand? How would I go about doing that?

Solution 1:

There is a simpler way to do this.

GCP offers a service called Cloud Functions. In them, you can put scripts that are called by Events or Triggers. One of them is Cloud Storage Triggers which are activated with Cloud Storage events, such as Object creation (upload files) or Object deletion (delete files).

The Cloud Function can read the file (in this case a Video), perform all the process in it and return a result or, in your case, store the result in a GCS bucket (could be the same where the file was uploaded, could be another one). Here you can find code samples to interact with the File you just uploaded.

To store your processed video, please check out the Cloud Storage Client libraries. Depending on what language you will use (the supported ones for Cloud Storage And Cloud Functions are Node.js, Python, Go, Java and C#) you should check for the code samples to Upload the result in a bucket.

Using Cloud Functions is a much cheaper option against Compute Engine as you will only use these resources when they are required. The only disadvantage here is that you are limited by the Memory and CPU this service supports.

If the resources (from 128MB to 4096MB in Memory) don't suit the process you want to perform, you can also follow these steps:

1. Create a Compute Engine instance with the Machine Type required for your use case. Put a script where all the process will be done. Once created and everything is setup, stop it.
2. Create a Cloud Function with GCS event triggers (with the already provided docs) and whenever it is called, start again your GCE via code. This can be done with the Compute Engine Client libraries or via the API (I can think of a HTTPS request with your favorite language).
3. Send all the information to your GCE and let the machine work (you might need to add some code to wait for an answer from GCE in Cloud Functions).
4. When it ends, upload the file to Cloud Storage and let Cloud Functions know that everything is done.
5. Once receiving this confirmation, tell Cloud Functions to stop your GCE again.

Remember that Compute Engine charges you for the time the VMs are running so doing this alternative scenario could also provide you the solution you want with the required resources in a reasonable pricing.

Scheduling a GCE with Cloud Scheduler is also an option but you don't really know when the GCE will be used, so using Cloud Functions will involve less products and less effort from your end.