How to create a BigQuery TransferConfig that overwrites the destination tables using the Python API
I want to automatically create a BigQuery DataTransfer using the Python Client API. I am already able to create it successfully and also to trigger it. However, my problem is that I want to set the flag overwrite_destination_table
to True. I can manually do that in the TransferConfig UI, however, I am not able to find any way of passing this flag via the Client API.
Question, does anybody know a way to set the flag overwrite_destination_table
for BigQuery DataTranfer in code?
You can specify the option using params
:
import os
import datetime
from google.cloud import bigquery_datatransfer, bigquery_datatransfer_v1
transfer_client = bigquery_datatransfer.DataTransferServiceClient()
project_id = os.getenv('PROJECT_ID')
dataset_id = os.getenv('BIGQUERY_DATASET')
service_account_name = os.getenv('BIGQUERY_SERVICE_ACCOUNT')
display_name = os.getenv('SCHEDULE_NAME')
query_string = """
SELECT * FROM demo_bq_dataset.demo_table;
"""
parent = transfer_client.common_project_path(project_id)
transfer_config = bigquery_datatransfer.TransferConfig(
destination_dataset_id=dataset_id,
display_name=display_name,
data_source_id="scheduled_query",
params={
"query": query_string,
"destination_table_name_template": "demo_destination_table",
"overwrite_destination_table": True,
},
schedule="every 4 hours synchronized",
schedule_options=bigquery_datatransfer_v1.types.ScheduleOptions(
start_time=datetime.datetime.now(datetime.timezone.utc)
)
)
transfer_config = transfer_client.create_transfer_config(
bigquery_datatransfer.CreateTransferConfigRequest(
parent=parent,
transfer_config=transfer_config,
service_account_name=service_account_name,
)
)
print("Created scheduled query '{}'".format(transfer_config.name))