Uly.me

cloud engineer

multiple

AWS S3 Sync Between Accounts

Here’s how to sync S3 buckets between 2 different AWS accounts. Assuming buckets are already created.

  1. Setup bucket permissions in Account A
  2. Setup IAM user with permissions in Account B
  3. Setup bucket permissions in Account B
  4. Run S3 sync from Account B.

Account A bucket permissions. Account and user are from Account B.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::222222222222:user/Jane"},
            "Action": ["s3:ListBucket","s3:GetObject"],
            "Resource": [
                "arn:aws:s3:::awsexamplesourcebucket/*",
                "arn:aws:s3:::awsexamplesourcebucket"
            ]
        }
    ]
}

Create IAM user (Jane) in Account B

aws iam create-user --user-name Jane

Give IAM user (Jane) access to both buckets.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::awsexamplesourcebucket",
                "arn:aws:s3:::awsexamplesourcebucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::awsexampledestinationbucket",
                "arn:aws:s3:::awsexampledestinationbucket/*"
            ]
        }
    ]
}

Sync the buckets

aws s3 sync s3://awsexamplesourcebucket s3://awsexampledestinationbucket

AWS CLI S3 Recursive Copy

Here’s how to copy multiple files recursively using AWS CLI. The S3 cp command by default only copies a single file. To copy multiple files, you have to use the –recursive option along with –exclude and –include. In this example, we will exclude every file, but include only files with a json extension.

aws s3 cp /tmp/folder s3://bucket/folder \
  --recursive
  --exclude "*"
  --include "*.json"

The result:

upload: ./abc.json to s3://bucket/folder/abc.json    
upload: ./xyz.json to s3://bucket/folder/xyz.json

GCP Start Multiple Instances

Here’s how to start multiple instances using Bash and Google SDK.

#!/bin/bash
while IFS=', ' read -r a b c; do
  instance="${a//[[:blank:]]/}"
  project="${b//[[:blank:]]/}"
  zone="${c//[[:blank:]]/}"
  echo "Starting $instance ...."
  gcloud compute instances start $instance \
  --zone $zone --project $project --async
done < instance-list.txt

Instances are read from an external file. The file format is displayed below.

server1,project1,us-central1-a
server2,project2,us-central1-b
server3,project3,us-central1-c

To stop multiple instances, replace “start” with “stop.”