• Skip to primary navigation
  • Skip to main content

Uly.me

cloud engineer

  • Home
  • About
  • Archives

bucket

AWS S3 Bucket Permission

by Ulysses · Feb 16, 2021

I was getting this error when downloading a file from a S3 bucket.

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Turns out to be a permission issue. Use –acl bucket-owner-full-control.

# UPLOAD
aws s3 cp file.txt s3://bucket-name/dir/ --acl bucket-owner-full-control
upload: .\file.txt to s3://bucket-name/dir/fw.sh
# DOWNLOAD
aws s3 cp s3://bucket-name/dir/file.txt . --acl bucket-owner-full-control
download: s3://bucket-name/dir/file.txt to .\file.txt

# UPLOAD aws s3 cp file.txt s3://bucket-name/dir/ --acl bucket-owner-full-control upload: .\file.txt to s3://bucket-name/dir/fw.sh # DOWNLOAD aws s3 cp s3://bucket-name/dir/file.txt . --acl bucket-owner-full-control download: s3://bucket-name/dir/file.txt to .\file.txt

You need to do for both upload and download.

Filed Under: Cloud Tagged With: aws, bucket, permissions, s3

GCP Bucket Size

by Ulysses · Mar 26, 2020

Here are the commands to find out the bucket size in GCP.

gsutil du -s gs://bucket-name/

gsutil du -s gs://bucket-name/

Filed Under: Cloud Tagged With: bucket, gcp, size

Standard S3 Policy

by Ulysses · Jan 28, 2020

Here’s a standard S3 policy to grant an IAM user access to a bucket within an AWS account. User is allowed to add, update, and delete objects. These 3 actions s3:ListAllMyBuckets, s3:GetBucketLocation, and s3:ListBucket are the additional permissions required to access the console. Also, the s3:PutObjectAcl and the s3:GetObjectAcl actions are required to be able to copy, cut, and paste objects within the console.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListAllMyBuckets"
         ],
         "Resource":"arn:aws:s3:::*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource":"arn:aws:s3:::examplebucket"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:DeleteObject"
         ],
         "Resource":"arn:aws:s3:::examplebucket/*"
      }
   ]
}

{ "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Action":[ "s3:ListAllMyBuckets" ], "Resource":"arn:aws:s3:::*" }, { "Effect":"Allow", "Action":[ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource":"arn:aws:s3:::examplebucket" }, { "Effect":"Allow", "Action":[ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject" ], "Resource":"arn:aws:s3:::examplebucket/*" } ] }

Filed Under: Cloud Tagged With: access, aws, bucket, grant, iam, policy, s3

AWS S3 Upload Policy

by Ulysses · Jul 26, 2019

Give someone upload access to a S3 bucket. Here’s the policy.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListAllMyBuckets"
         ],
         "Resource":"arn:aws:s3:::*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource":"arn:aws:s3:::your-bucket-name"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:DeleteObject"
         ],
         "Resource":"arn:aws:s3:::your-bucket-name/*"
      }
   ]
}

{ "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Action":[ "s3:ListAllMyBuckets" ], "Resource":"arn:aws:s3:::*" }, { "Effect":"Allow", "Action":[ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource":"arn:aws:s3:::your-bucket-name" }, { "Effect":"Allow", "Action":[ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject" ], "Resource":"arn:aws:s3:::your-bucket-name/*" } ] }

Filed Under: Cloud Tagged With: aws, bucket, policy, s3, upload

S3 Restrict IP Addresses

by Ulysses · Jun 5, 2019

Here’s the policy to restrict access to S3 bucket to certain IP addresses.

{
    "Version": "2012-10-17",
    "Id": "S3PolicyIPRestrict",
    "Statement": [
        {
            "Sid": "IPAllow",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*" 
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::bucket/*",
            "Condition" : {
                "IpAddress" : {
                    "aws:SourceIp": "10.10.10.0/24" 
                },
                "NotIpAddress" : {
                    "aws:SourceIp": "10.10.10.100/32" 
                } 
            } 
        } 
    ]
}

{ "Version": "2012-10-17", "Id": "S3PolicyIPRestrict", "Statement": [ { "Sid": "IPAllow", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:*", "Resource": "arn:aws:s3:::bucket/*", "Condition" : { "IpAddress" : { "aws:SourceIp": "10.10.10.0/24" }, "NotIpAddress" : { "aws:SourceIp": "10.10.10.100/32" } } } ] }

Allow anyone in the 10.10.10.0/24 network except for 10.10.10.100/32.

Filed Under: Cloud Tagged With: aws, bucket, ip address, policy, restric, s3

AWS Glacier Setup via S3

by Ulysses · May 13, 2019

You can setup AWS Glacier via S3 bucket replication. Create a S3 bucket and slap this bucket policy.

{
    "Version": "2012-10-17",
    "Id": "S3PolicyId1",
    "Statement": [
        {
            "Sid": "IPAllow",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::your-bucket-storage-name/*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "10.0.0.0/8"
                }
            }
        },
        {
            "Sid": "DenyIncorrectEncryptionHeader",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::your-bucket-storage-name/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::your-bucket-storage-name/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}

{ "Version": "2012-10-17", "Id": "S3PolicyId1", "Statement": [ { "Sid": "IPAllow", "Effect": "Allow", "Principal": "*", "Action": "s3:*", "Resource": "arn:aws:s3:::your-bucket-storage-name/*", "Condition": { "IpAddress": { "aws:SourceIp": "10.0.0.0/8" } } }, { "Sid": "DenyIncorrectEncryptionHeader", "Effect": "Deny", "Principal": "*", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::your-bucket-storage-name/*", "Condition": { "StringNotEquals": { "s3:x-amz-server-side-encryption": "AES256" } } }, { "Sid": "DenyUnEncryptedObjectUploads", "Effect": "Deny", "Principal": "*", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::your-bucket-storage-name/*", "Condition": { "Null": { "s3:x-amz-server-side-encryption": "true" } } } ] }

Add this policy to your IAM user or role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::cah-callcopy-storage/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*"
            ],
            "Resource": "arn:aws:s3:::*"
        }
    ]
}

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": "arn:aws:s3:::cah-callcopy-storage/*" }, { "Effect": "Allow", "Action": [ "s3:List*", "s3:Get*" ], "Resource": "arn:aws:s3:::*" } ] }

Finally, add a lifecycle policy to move your files from Standard to Glacier storage type.

Filed Under: Cloud Tagged With: aws, bucket, glacier, iam, policy, role, s3, user

AWS CLI Create Bucket

by Ulysses · Dec 9, 2018

Creating a bucket via the AWS Console takes only a few clicks. If you’re creating a bucket via shell script, you will need to use the AWS CLI (command line interface). So, here’s the script to create a S3 bucket via the AWS CLI within shell. Now the bucket name has to be globally unique similar to a domain name. If someone already has claimed it, you will need to use an alternative name.

aws s3api create-bucket --bucket bucket-name --region us-east-2 create-bucket-configuration LocationConstraint=us-east-2

aws s3api create-bucket --bucket bucket-name --region us-east-2 create-bucket-configuration LocationConstraint=us-east-2

Filed Under: Cloud, Linux Tagged With: aws cli, bucket, create, s3

Copyright © 2012–2021