• Skip to main content

Uly.me

cloud engineer

  • Home
  • About
  • Archives

Copy S3 to GCS

October 13, 2021

You can use gsutil to copy directly from a S3 bucket to a GCS bucket.

gsutil can read/write to a S3 bucket as long as it has access to your AWS credentials.

The -R option is recursive and -m is for multi-threaded or multi-processing.

gsutil -m cp -R s3://bucket/ gs://bucket

gsutil -m cp -R s3://bucket/ gs://bucket

You can also use rsync instead of cp. Just be aware that rsync does not copy empty directory trees

gsutil -m rsync -r s3://bucket/ gs://bucket

gsutil -m rsync -r s3://bucket/ gs://bucket

If you plan to use -d, Use it with caution since it deletes content.

You may have to set the following to your defaults accounts.

export AWS_PROFILE=yourprofile
gcloud config set project your-project-id

export AWS_PROFILE=yourprofile gcloud config set project your-project-id

This ensures that default profile and projects are used in both AWS and GCP.

Filed Under: Cloud Tagged With: aws, cp, gcp, gcs, multi-processing, multi-threaded, rsync, s3

Search This Website

Subscribe Via Email

  • Home
  • About
  • Archives

Copyright © 2023