Uly.me

cloud engineer

bucket

GCS Fuse

GCS Fuse allows you to mount a Google bucket as a file system. It’s similar to S3FS.

Setup repo

sudo tee /etc/yum.repos.d/gcsfuse.repo > /dev/null <<EOF
[gcsfuse]
name=gcsfuse (packages.cloud.google.com)
baseurl=https://packages.cloud.google.com/yum/repos/gcsfuse-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Yum install

sudo yum install gcsfuse

Login to GCP and mount. Run as a user and not root.

gcloud auth login
gcsfuse my-bucket /path/to/mount

Unmount

fusermount -u /path/to/mount

Terraform AWS S3

How to create S3 bucket via Terraform.

erraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}
 
provider "aws" {
  profile = "default"
  region  = "us-east-1"
}
 
resource "aws_s3_bucket" "bucket" {
  bucket = "my-ulysses-bucket"
  acl    = "private"
 
  tags = {
    Name        = "My Ulysses bucket"
    Environment = "Dev"
  }
}
 
resource "aws_s3_bucket_public_access_block" "example" {
  bucket = aws_s3_bucket.bucket.id
  block_public_acls = true
  block_public_policy = true
  ignore_public_acls = true
  restrict_public_buckets = true
}

S3FS

s3fs allows Linux to mount S3 buckets as a file system.

Install s3fs.

sudo apt install s3fs

Setup credentials.

echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > /etc/.passwd-s3fs
chmod 600 ${HOME}/.passwd-s3fs

Mount it.

s3fs bucketname /mountpoint -o passwd_file=/etc/.passwd-s3fs

Mount it automatically.

bucketname /mountpoint fuse.s3fs _netdev,allow_other,passwd_file=/etc/.passwd-s3fs,rw,uid=1000,gid=1000 0 0

Alternative.

s3fs#bucketname /mountpoint fuse _netdev,allow_other,use_cache=/root/cache,uid=1000,gid=1000,umask=022 0 0

AWS S3 Bucket Permission

I was getting this error when downloading a file from a S3 bucket.

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Turns out to be a permission issue. Use –acl bucket-owner-full-control.

# UPLOAD
aws s3 cp file.txt s3://bucket-name/dir/ --acl bucket-owner-full-control
upload: .\file.txt to s3://bucket-name/dir/fw.sh
# DOWNLOAD
aws s3 cp s3://bucket-name/dir/file.txt . --acl bucket-owner-full-control
download: s3://bucket-name/dir/file.txt to .\file.txt

You need to do for both upload and download.

GCP Bucket Size

Here are the commands to find out the bucket size in GCP.

gsutil du -s gs://bucket-name/

Standard S3 Policy

Here’s a standard S3 policy to grant an IAM user access to a bucket within an AWS account. User is allowed to add, update, and delete objects. These 3 actions s3:ListAllMyBuckets, s3:GetBucketLocation, and s3:ListBucket are the additional permissions required to access the console. Also, the s3:PutObjectAcl and the s3:GetObjectAcl actions are required to be able to copy, cut, and paste objects within the console. 

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListAllMyBuckets"
         ],
         "Resource":"arn:aws:s3:::*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource":"arn:aws:s3:::examplebucket"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:DeleteObject"
         ],
         "Resource":"arn:aws:s3:::examplebucket/*"
      }
   ]
}

AWS S3 Upload Policy

Give someone upload access to a S3 bucket. Here’s the policy.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListAllMyBuckets"
         ],
         "Resource":"arn:aws:s3:::*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource":"arn:aws:s3:::your-bucket-name"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:DeleteObject"
         ],
         "Resource":"arn:aws:s3:::your-bucket-name/*"
      }
   ]
}