Uly.me

cloud engineer

s3

AWS S3 Bucket Permission

by ·

I was getting this error when downloading a file from a S3 bucket.

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Turns out to be a permission issue. Use –acl bucket-owner-full-control.

# UPLOAD
aws s3 cp file.txt s3://bucket-name/dir/ --acl bucket-owner-full-control
upload: .\file.txt to s3://bucket-name/dir/fw.sh
# DOWNLOAD
aws s3 cp s3://bucket-name/dir/file.txt . --acl bucket-owner-full-control
download: s3://bucket-name/dir/file.txt to .\file.txt

You need to do for both upload and download.

AWS S3 Make Object Public

by ·

Copy object or file to S3 bucket.

aws s3 cp filename.ext s3://bucketname/ --profile your-profile

To make it publicly available, run this command.

aws s3api put-object-acl \
--bucket bucket-name \
--key filename.ext \
--acl public-read \
--profile your-profile

AWS S3 Acceleration CLI

by ·

Here is how to enable Amazon S3 Transfer Acceleration on a S3 bucket.

aws s3api put-bucket-accelerate-configuration \
--bucket bucketname \
--accelerate-configuration Status=Enabled \
--region us-east-1

Use an accelerate endpoint.

aws configure set default.s3.use_accelerate_endpoint true

To copy files to S3 you can use the default copy.

aws s3 cp file.txt s3://bucketname/keyname \
--region us-east-1

Or use the acceleration endpoint.

aws configure set s3.addressing_style virtual
aws s3 cp file.txt s3://bucketname/keyname \
--endpoint-url http://s3-accelerate.amazonaws.com \
--region us-east-1

AWS S3 Sync Between Accounts

by ·

Here’s how to sync S3 buckets between 2 different AWS accounts. Assuming buckets are already created.

  1. Setup bucket permissions in Account A
  2. Setup IAM user with permissions in Account B
  3. Setup bucket permissions in Account B
  4. Run S3 sync from Account B.

Account A bucket permissions. Account and user are from Account B.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::222222222222:user/Jane"},
            "Action": ["s3:ListBucket","s3:GetObject"],
            "Resource": [
                "arn:aws:s3:::awsexamplesourcebucket/*",
                "arn:aws:s3:::awsexamplesourcebucket"
            ]
        }
    ]
}

Create IAM user (Jane) in Account B

aws iam create-user --user-name Jane

Give IAM user (Jane) access to both buckets.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::awsexamplesourcebucket",
                "arn:aws:s3:::awsexamplesourcebucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::awsexampledestinationbucket",
                "arn:aws:s3:::awsexampledestinationbucket/*"
            ]
        }
    ]
}

Sync the buckets

aws s3 sync s3://awsexamplesourcebucket s3://awsexampledestinationbucket

S3 Browser IAM Assume Role

by ·

S3 Browser is a free Windows client for accessing AWS S3 buckets. You can access S3 buckets with an interface similar to File Manager. A typical setup requires that you use access and secret keys to connect to the S3 browser. However, if the server is in the cloud, it’s better that use a machine profile with an assume role. The assume role gets temporary credentials until they expire. They are automatically rotated by AWS behind the scenes for you. This is much safer process than storing key credentials in the cloud. The good news is, S3 Browser support IAM assume roles. Here’s the setup instructions to get your S3 Browser configured using IAM assume role. Here’s IAM role explained.

AWS S3 LS Recursive

by ·

Here’s how to list S3 files recursively.

aws s3 ls s3://mybucket --recursive | awk '{print $4}'

By default, the results display date, time, disk size and filename. Use awk to display the filename only.

AWS CLI S3 Recursive Copy

by ·

Here’s how to copy multiple files recursively using AWS CLI. The S3 cp command by default only copies a single file. To copy multiple files, you have to use the –recursive option along with –exclude and –include. In this example, we will exclude every file, but include only files with a json extension.

aws s3 cp /tmp/folder s3://bucket/folder \
  --recursive
  --exclude "*"
  --include "*.json"

The result:

upload: ./abc.json to s3://bucket/folder/abc.json    
upload: ./xyz.json to s3://bucket/folder/xyz.json

Standard S3 Policy

by ·

Here’s a standard S3 policy to grant an IAM user access to a bucket within an AWS account. User is allowed to add, update, and delete objects. These 3 actions s3:ListAllMyBuckets, s3:GetBucketLocation, and s3:ListBucket are the additional permissions required to access the console. Also, the s3:PutObjectAcl and the s3:GetObjectAcl actions are required to be able to copy, cut, and paste objects within the console. 

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListAllMyBuckets"
         ],
         "Resource":"arn:aws:s3:::*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource":"arn:aws:s3:::examplebucket"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:DeleteObject"
         ],
         "Resource":"arn:aws:s3:::examplebucket/*"
      }
   ]
}

AWS Boto3 Client

by ·

Boto is the Amazon Web Services (AWS) SDK for Python. It enables Python developers to create, configure, and manage AWS services, such as EC2 and S3. Boto provides an easy to use, object-oriented API, as well as low-level access to AWS services.

Here’s a quick example how to query for AWS resources.

import logging
import boto3
from botocore.exceptions import ClientError
 
# set aws profile
session = boto3.Session(profile_name="default")
 
# get buckets
s3 = session.client('s3')
results = s3.list_buckets()
 
# display buckets
print('Existing buckets:')
for key in results['Buckets']:
    print(f'{key["Name"]}')
 
# get ec2 instances. set region.
ec2 = session.client('ec2', region_name='us-east-1')
results = ec2.describe_instances()
 
# display instances
for key in results["Reservations"]:
      for instance in key["Instances"]:
           instance_id = instance["InstanceId"]
           instance_type = instance["InstanceType"]
           availability_zone = instance["Placement"]["AvailabilityZone"]
           print(instance_id + "\t" + instance_type + "\t" + availability_zone)
 
# get list of users
client = session.client('iam')
results = client.list_users()
 
# display list of users
for key in results['Users']:
    print (key['UserName'])

AWS S3 Replication Policy

by ·

Here’s the policy for S3 replication between regions.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Get*",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::source-bucket-name",
                "arn:aws:s3:::source-bucket-name/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::destination-bucket-name/*"
        }
    ]
}