How can I upload folders to Google Cloud?

How can I upload complete folders to my Google Cloud Storage? I tried using Cyberduck, but I can't get my Google login info to work with it. Any ideas for a medium-skill level Mac user to easily upload complete folders rather than files?

Solution 1:

You can use gsutil cp in the Terminal for bulk uploads/downloads:

gsutil -m cp -R dir gs://my_bucket

The -m flag will perform a parallel (multi-threaded) copy.

Solution 2:

You can easily upload folders from your local computer using the Google Cloud Console with no special tools.

Just click the left hand navigation Storage > Cloud Storage > Browser using in the Google Cloud Console: https://console.developers.google.com

Create a bucket or select an existing one

Then click "Upload folder"

You'll get a sidebar that will pop out notifying you of the upload progress.

Solution 3:

Just for completeness, I show how to:

1. Get started (login, set project)
2. Upload the files
3. List the items
4. Download the files An alternative to cp is to use rsync which is not demonstrated here.

Start the session, set the project and see the current configuration:

gcloud auth login
gcloud config set project PROJECT_ID
gcloud config list

Create test tree folder structure to upload

echo 'test content: foo BAR" > ec2-iamrole.yaml
mkdir -p level1/level2
cp ec2-iamrole.yaml level1
cp ec2-iamrole.yaml level1/level2
rm ec2-iamrole.yaml level1

To upload (bucket was already created and it has a folder name sample)

gsutil -m cp -R level1 gs://my_bucket/sample

To create the bucket, check here. In addition, you cannot create single folders using gsutil command. You need to use the GCP UI or user the client library or explicitly specifying it in the command as I just did above.

List content:

gsutil ls -r gs://my_bucket/sample/level1

Output:

gs://my_bucket/sample/level1/:
gs://my_bucket/sample/level1/ec2-iamrole.yaml

gs://my_bucket/sample/level1/level2/:
gs://my_bucket/sample/level1/level2/ec2-iamrole.yaml

Download into a new folder level3:

mkdir level3 && cd level3
gsutil -m cp -R gs://docexporter/nov/level1 .

Output:

tree .
.
└── level1
    ├── ec2-iamrole.yaml
    └── level2
        └── ec2-iamrole.yaml

2 directories, 2 files