cloud engineer
You can use gsutil to copy directly from a S3 bucket to a GCS bucket.
gsutil can read/write to a S3 bucket as long as it has access to your AWS credentials.
The -R option is recursive and -m is for multi-threaded or multi-processing.
gsutil -m cp -R s3://bucket/ gs://bucket |
gsutil -m cp -R s3://bucket/ gs://bucket
You can also use rsync instead of cp. Just be aware that rsync does not copy empty directory trees
gsutil -m rsync -r s3://bucket/ gs://bucket |
gsutil -m rsync -r s3://bucket/ gs://bucket
If you plan to use -d, Use it with caution since it deletes content.
Buckets are regional. Your source and destination buckets should be in different regions.
aws s3 sync s3://DOC-EXAMPLE-BUCKET-SOURCE s3://DOC-EXAMPLE-BUCKET-TARGET |
aws s3 sync s3://DOC-EXAMPLE-BUCKET-SOURCE s3://DOC-EXAMPLE-BUCKET-TARGET
Use the sync command. It will copy new or modified files.
s3fs allows Linux to mount S3 buckets as a file system.
Install s3fs.
sudo apt install s3fs |
sudo apt install s3fs
Setup credentials.
echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs chmod 600 ${HOME}/.passwd-s3fs |
echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs chmod 600 ${HOME}/.passwd-s3fs
Mount it.
s3fs bucketname /mountpoint -o passwd_file=${HOME}/.passwd-s3fs |
s3fs bucketname /mountpoint -o passwd_file=${HOME}/.passwd-s3fs
Mount it automatically.
bucketname /mountpoint fuse.s3fs _netdev,allow_other,passwd_file=/home/user/.passwd-s3fs,rw,uid=1000,gid=1000 0 0 |
bucketname /mountpoint fuse.s3fs _netdev,allow_other,passwd_file=/home/user/.passwd-s3fs,rw,uid=1000,gid=1000 0 0
I was getting this error when downloading a file from a S3 bucket.
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden |
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
Turns out to be a permission issue. Use –acl bucket-owner-full-control.
# UPLOAD aws s3 cp file.txt s3://bucket-name/dir/ --acl bucket-owner-full-control upload: .\file.txt to s3://bucket-name/dir/fw.sh # DOWNLOAD aws s3 cp s3://bucket-name/dir/file.txt . --acl bucket-owner-full-control download: s3://bucket-name/dir/file.txt to .\file.txt |
# UPLOAD aws s3 cp file.txt s3://bucket-name/dir/ --acl bucket-owner-full-control upload: .\file.txt to s3://bucket-name/dir/fw.sh # DOWNLOAD aws s3 cp s3://bucket-name/dir/file.txt . --acl bucket-owner-full-control download: s3://bucket-name/dir/file.txt to .\file.txt
You need to do for both upload and download.
Copy object or file to S3 bucket.
aws s3 cp filename.ext s3://bucketname/ --profile your-profile |
aws s3 cp filename.ext s3://bucketname/ --profile your-profile
To make it publicly available, run this command.
aws s3api put-object-acl \ --bucket bucket-name \ --key filename.ext \ --acl public-read \ --profile your-profile |
aws s3api put-object-acl \ --bucket bucket-name \ --key filename.ext \ --acl public-read \ --profile your-profile