Uly.me

cloud engineer

s3

AWS S3 Sync Between Accounts

Here’s how to sync S3 buckets between 2 different AWS accounts. Assuming buckets are already created.

  1. Setup bucket permissions in Account A
  2. Setup IAM user with permissions in Account B
  3. Setup bucket permissions in Account B
  4. Run S3 sync from Account B.

Account A bucket permissions. Account and user are from Account B.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::222222222222:user/Jane"},
            "Action": ["s3:ListBucket","s3:GetObject"],
            "Resource": [
                "arn:aws:s3:::awsexamplesourcebucket/*",
                "arn:aws:s3:::awsexamplesourcebucket"
            ]
        }
    ]
}

Create IAM user (Jane) in Account B

aws iam create-user --user-name Jane

Give IAM user (Jane) access to both buckets.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::awsexamplesourcebucket",
                "arn:aws:s3:::awsexamplesourcebucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::awsexampledestinationbucket",
                "arn:aws:s3:::awsexampledestinationbucket/*"
            ]
        }
    ]
}

Sync the buckets

aws s3 sync s3://awsexamplesourcebucket s3://awsexampledestinationbucket

S3 Browser IAM Assume Role

S3 Browser is a free Windows client for accessing AWS S3 buckets. You can access S3 buckets with an interface similar to File Manager. A typical setup requires that you use access and secret keys to connect to the S3 browser. However, if the server is in the cloud, it’s better that use a machine profile with an assume role. The assume role gets temporary credentials until they expire. They are automatically rotated by AWS behind the scenes for you. This is much safer process than storing key credentials in the cloud. The good news is, S3 Browser support IAM assume roles. Here’s the setup instructions to get your S3 Browser configured using IAM assume role. Here’s IAM role explained.

AWS S3 LS Recursive

Here’s how to list S3 files recursively.

aws s3 ls s3://mybucket --recursive | awk '{print $4}'

By default, the results display date, time, disk size and filename. Use awk to display the filename only.

AWS CLI S3 Recursive Copy

Here’s how to copy multiple files recursively using AWS CLI. The S3 cp command by default only copies a single file. To copy multiple files, you have to use the –recursive option along with –exclude and –include. In this example, we will exclude every file, but include only files with a json extension.

aws s3 cp /tmp/folder s3://bucket/folder \
  --recursive
  --exclude "*"
  --include "*.json"

The result:

upload: ./abc.json to s3://bucket/folder/abc.json    
upload: ./xyz.json to s3://bucket/folder/xyz.json

Standard S3 Policy

Here’s a standard S3 policy to grant an IAM user access to a bucket within an AWS account. User is allowed to add, update, and delete objects. These 3 actions s3:ListAllMyBuckets, s3:GetBucketLocation, and s3:ListBucket are the additional permissions required to access the console. Also, the s3:PutObjectAcl and the s3:GetObjectAcl actions are required to be able to copy, cut, and paste objects within the console. 

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListAllMyBuckets"
         ],
         "Resource":"arn:aws:s3:::*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource":"arn:aws:s3:::examplebucket"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:DeleteObject"
         ],
         "Resource":"arn:aws:s3:::examplebucket/*"
      }
   ]
}

AWS Boto3 Client

Boto is the Amazon Web Services (AWS) SDK for Python. It enables Python developers to create, configure, and manage AWS services, such as EC2 and S3. Boto provides an easy to use, object-oriented API, as well as low-level access to AWS services.

Here’s a quick example how to query for AWS resources.

import logging
import boto3
from botocore.exceptions import ClientError
 
# set aws profile
session = boto3.Session(profile_name="default")
 
# get buckets
s3 = session.client('s3')
results = s3.list_buckets()
 
# display buckets
print('Existing buckets:')
for key in results['Buckets']:
    print(f'{key["Name"]}')
 
# get ec2 instances. set region.
ec2 = session.client('ec2', region_name='us-east-1')
results = ec2.describe_instances()
 
# display instances
for key in results["Reservations"]:
      for instance in key["Instances"]:
           instance_id = instance["InstanceId"]
           instance_type = instance["InstanceType"]
           availability_zone = instance["Placement"]["AvailabilityZone"]
           print(instance_id + "\t" + instance_type + "\t" + availability_zone)
 
# get list of users
client = session.client('iam')
results = client.list_users()
 
# display list of users
for key in results['Users']:
    print (key['UserName'])