Uly.me

cloud engineer

s3

Python Boto3 S3 List Objects

How to list S3 objects using Python Boto3

import boto3
from sys import argv
bucket=argv[1]
s3 = boto3.resource('s3')
mybucket = s3.Bucket(bucket)
def list_buckets():
    print("List of Buckets: ")
    for bucket in s3.buckets.all():
        print(bucket.name)
def list_files():
    print("Buckets: The objects in the \"" + bucket + "\" buckets are ... ")
    for files in mybucket.objects.all():
        print(files.key)
def main():
    #list_buckets()
    list_files()
if __name__ == "__main__":
    main()

Running command

$ python test.py bucket-name    # with python
$ ./test.py bucket-name         # without python

Results

Buckets: The objects in the "bucket-name" are ....
abc.txt
def.txt

Terraform AWS S3

How to create S3 bucket via Terraform.

erraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}
 
provider "aws" {
  profile = "default"
  region  = "us-east-1"
}
 
resource "aws_s3_bucket" "bucket" {
  bucket = "my-ulysses-bucket"
  acl    = "private"
 
  tags = {
    Name        = "My Ulysses bucket"
    Environment = "Dev"
  }
}
 
resource "aws_s3_bucket_public_access_block" "example" {
  bucket = aws_s3_bucket.bucket.id
  block_public_acls = true
  block_public_policy = true
  ignore_public_acls = true
  restrict_public_buckets = true
}

AWS Set Profile

If you find yourself using a named profile for every awscli command, you can set it temporarily.

Listing a bucket with a named profile.

aws s3 ls --profile=yourprofile

If you set AWS_PROFILE, you can then list a bucket without a named profile.

export AWS_PROFILE=yourprofile
aws s3 ls

After you are done, you can reset it back to default.

export AWS_PROFILE=default

Copy S3 to GCS

You can use gsutil to copy directly from a S3 bucket to a GCS bucket.

gsutil can read/write to a S3 bucket as long as it has access to your AWS credentials.

The -R option is recursive and -m is for multi-threaded or multi-processing.

gsutil -m cp -R s3://bucket/ gs://bucket

You can also use rsync instead of cp. Just be aware that rsync does not copy empty directory trees

gsutil -m rsync -r s3://bucket/ gs://bucket

If you plan to use -d, Use it with caution since it deletes content.

You may have to set the following to your defaults accounts.

export AWS_PROFILE=yourprofile
gcloud config set project your-project-id

This ensures that default profile and projects are used in both AWS and GCP.

Copy S3 To Another Region

Buckets are regional. Your source and destination buckets should be in different regions.

aws s3 sync s3://DOC-EXAMPLE-BUCKET-SOURCE s3://DOC-EXAMPLE-BUCKET-TARGET

Use the sync command. It will copy new or modified files.