cloud engineer
Here’s how to list AWS AMI (machine images) owned by yourself.
aws ec2 describe-images \ --query "Images[*].[Name,PlatformDetails]" \ --owners self \ --profile default \ --region us-east-1 \ --output text |
Output:
icecast-0.1 Linux/UNIX jekyll-0.3 Linux/UNIX docker-0.1 Linux/UNIX |
Specifying owners as self only display images owned by you.
I was getting this error when downloading a file from a S3 bucket.
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden |
Turns out to be a permission issue. Use –acl bucket-owner-full-control.
# UPLOAD aws s3 cp file.txt s3://bucket-name/dir/ --acl bucket-owner-full-control upload: .\file.txt to s3://bucket-name/dir/fw.sh # DOWNLOAD aws s3 cp s3://bucket-name/dir/file.txt . --acl bucket-owner-full-control download: s3://bucket-name/dir/file.txt to .\file.txt |
You need to do for both upload and download.
Here’s how to create a firewall from service account to service account.
gcloud compute firewall-rules create "firewall-name" \ --description="firewall-description" \ --priority "1000" \ --direction INGRESS \ --action allow \ --network "network-name" \ --source-service-accounts="service@account.net" \ --target-service-accounts="service@account.net" \ --rules tcp:9001 |
Instead of source-range, it’s using source-service-accounts.
I have an AMI with docker installed. Here’s how I launch a spot instance using Terraform.
Here’s my Terraform script.
terraform { required_providers { aws = { source = "hashicorp/aws" } } } provider "aws" { profile = "default" region = "us-east-1" } resource "aws_spot_instance_request" "docker" { ami = "ami-xxxxxxxxxxxxxxxx" spot_price = "0.0031" wait_for_fulfillment = "true" key_name = "servers" instance_type = "t3.micro" subnet_id = "subnet-xxxxxxxxxxxxxxxx" security_groups = ["sg-xxxxxxxxxxxxxxxxxx"] associate_public_ip_address = "true" user_data = <<-EOF #!/bin/bash hostnamectl set-hostname docker EOF tags = { Name = "docker-0.1" } } resource "aws_ec2_tag" "tagging" { resource_id = aws_spot_instance_request.docker.spot_instance_id key = "Name" value = "docker-0.1" } |
I use “aws_ec2_tag” resource to tag the instance properly.
In addition, I use user_data to run a script, to set the hostname.
To launch via Terraform, I run the following commands.
terraform init terraform apply |
When done, I could stop the instance to stop incurring charges. Or just simply destroy it via Terraform.
terraform destroy |
It’s not bad deal for an instance that costs only $0.0031 per hour.
Here’s a Terraform script for launching an instance in Google Cloud Platform.
provider "google" { project = "project-id" region = "us-central1" zone = "us-central1-a" } resource "google_compute_instance" "wiki" { name = "wiki" machine_type = "n2-standard-1" zone = "us-central1-a" tags = ["web-server"] labels = { name = "wiki" environment = "development" } boot_disk { initialize_params { image = "centos-7-v20210122" } } network_interface { network = "default" access_config { } } service_account { email = "service-account@email.com" scopes = ["cloud-platform"] } } |
Here’s a few Terraform commands.
terraform init terraform plan terraform apply terraform destroy |
If you have multiple AWS accounts, you can setup a customer-managed KMS (key management service) in the AWS Key Management Service, to secure requests or services between the two AWS accounts. The customer-managed KMS key is tied to an identity such as an IAM user or role. In addition to users and roles, other AWS accounts can be added to grant access. KMS can be symmetric or asymmetric. It’s symmetric be default. To grant access to the other account, you need to add the AWS Account Id to the key. It’s 12 digit number unique to each AWS account.
Once a key is created, the valid key ID can be used in a AWS SDK to access resources from the other AWS account.
AWS EC2 Firewall rules are defined within security groups. Security groups are attached to an instance. An instance can have up to 5 security groups. Essentially, this script gathers all the security groups associated with an instance, loops through them, and then outputs the ingress and egress rules of each security group to a file in a text format.
#!/bin/bash # set variables instanceid='i-xxxxxxxxxxxxxxxx' region='us-east-1' profile='sample' # log and temp files output="ec2-sg.log" tmpfil="ec2-sg.tmp" # empty log at start > $output # get sg ids aws ec2 describe-instances \ --instance-ids $instanceid \ --region $region \ --profile $profile \ --query 'Reservations[*].Instances[*].SecurityGroups[*].[GroupId]' --output text > $tmpfil while read -r id; do echo '============================================' >> $output echo $id >> $output echo '============================================' >> $output echo '---------------- INGRESS -------------------' >> $output aws ec2 describe-security-groups \ --group-ids $id \ --profile $profile \ --region $region \ --output text \ --query 'SecurityGroups[].IpPermissions[].[FromPort,ToPort,IpProtocol,IpRanges[].CidrIp[]|[0]]' >> $output echo '---------------- EGRESS --------------------' >> $output aws ec2 describe-security-groups \ --group-ids $id \ --profile $profile \ --region $region \ --output text \ --query 'SecurityGroups[].IpPermissionsEgress[].[FromPort,ToPort,IpProtocol,IpRanges[].CidrIp[]|[0]]' >> $output done < $tmpfil |
Here’s a sample output.
============================================ sg-xxxxxxxxxxxxxxx ============================================ ---------------- INGRESS ------------------- 5985 5985 tcp 10.0.0.220/32 10005 10005 tcp 10.0.0.164/32 ---------------- EGRESS -------------------- 80 80 tcp 10.0.0.14/32 40000 65535 udp 10.0.0.0/8 3389 3389 tcp 10.0.0.96/32 9389 9389 tcp 10.0.0.0/8 5985 5986 tcp 10.0.0.96/32 |
Here’s how to launch a LightSail instance using Terraform. Create a main.tf file.
terraform { required_providers { aws = { version = >= 3.22.0" source = "hashicorp/aws" } } } provider "aws" { profile = "default" region = "us-east-1" } resource "aws_lightsail_instance" "yourinstance" { name = "yourinstance" availability_zone = "us-east-1a" blueprint_id = "amazon_linux_2" bundle_id = "nano_2_0" } |
To launch, run the following Terraform commands.
terraform init terraform apply |
Here’s the script how to install Docker on Ubuntu 20.04 LTS.
#!/bin/bash sudo apt-get update # install dependencies sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common # add the gpg key for docker curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - # add the repository in the Linux mint 20 sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo "$UBUNTU_CODENAME") stable" sudo apt-get update # install docker e docker-compose sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo "$UBUNTU_CODENAME") stable" sudo apt-get -y install docker-ce docker-compose # add the user system to sudo group, no sudo command sudo usermod -aG docker $USER # prints docker version =) docker --version |
Here’s how to build an EFS file system using Terraform. Create a main.tf file.
terraform { required_providers { aws = { version = ">= 3.22.0" source = "hashicorp/aws" } } } provider "aws" { profile = "default" region = "us-east-1" } resource "aws_efs_file_system" "efs-test" { creation_token = "efs-test" performance_mode = "generalPurpose" throughput_mode = "bursting" encrypted = "true" tags = { Name = "efs-test" } } resource "aws_efs_mount_target" "efs-mt-example" { file_system_id = aws_efs_file_system.efs-test.id subnet_id = "subnet-xxxxxxxxxxxxxxxxx" security_groups = ["sg-xxxxxxxxxxxxxxxxxx"] } |
To launch, run terraform.
terraform init terraform apply |