How large a file size from GCS can handle in Cloud Function? [closed]

The first sentence in your question:

I want to transfer a >=4 GB of data from GCS to BigQuery(BQ)

If we stop right there, you do not need Cloud Function(CF) for the transfer. BQ is totally capable of taking care of ingesting your BIG CSV file from GCS (up to 15TB). But you need to trigger this ingestion or more precisely the BQ load job. Here enters Cloud Function. It can be plugged to your bucket. As soon as your file finishes uploading to GCS it will trigger the cloud function that will run the BigQuery Load job.

Below an example of load job to run with CF in python39. Dont forget to set the table_id And for more information or other languages, you can check this doc.

# the file should be named main.py

def load_csv_from_gcs_to_bq(data, _):
    file_name = data["name"]
    bucket_name = data["bucket"]
    # Construct the GCS file uri to load
    uri = f"gs://{bucket_name}/{file_name}"

    from google.cloud import bigquery

    # Construct a BigQuery client object.
    client = bigquery.Client()

    # TODO: Set table_id to the ID of the table to create.
    # table_id = "your-project.your_dataset.your_table_name"


    job_config = bigquery.LoadJobConfig(
        autodetect=True,
        skip_leading_rows=1,
        source_format=bigquery.SourceFormat.CSV,
    )

    load_job = client.load_table_from_uri(
        uri, table_id, job_config=job_config
    )  # Make an API request.

    load_job.result()  # Waits for the job to complete.

    destination_table = client.get_table(table_id)  # Make an API request.
    print("Loaded {} rows.".format(destination_table.num_rows))

With the requirements.txt file

google-cloud-bigquery==2.24.0

And here the command to deploy the CF (set YOUR_GCS_BUCKET without gs://) Check this link for more details.

gcloud functions deploy load_csv_from_gcs_to_bq \
--region europe-west1 \
--runtime python39 \
--timeout 300 \
--memory 128 \
--trigger-resource <YOUR_GCS_BUCKET> \
--trigger-event google.storage.object.finalize

Once your CSV is loaded in BQ, you can use BQ to do many types of transformation and store the result in new table(s).

But If you need custom and very specific transformation, for the size of your data Dataflow will be a better choice.