Uly.me

cloud engineer

output

Bash Read File

How to read a file line by line in Bash.

#!/bin/bash
file=test.txt
while IFS= read -r line; do
  echo $line
done < $file

Contents of test.txt.

one
two
three

Results when running test.sh.

$ bash test.sh
one
two
three

To read multiple strings line by line.

#!/bin/bash
file=test.txt
while read -r a b; do
  echo $b
done < $file

Modified contents of test.txt.

one 1
two 2
three 3

New results.

$ bash test.sh
1
2
3

AWS EC2 List Firewall Rules

AWS EC2 Firewall rules are defined within security groups. Security groups are attached to an instance. An instance can have up to 5 security groups. Essentially, this script gathers all the security groups associated with an instance, loops through them, and then outputs the ingress and egress rules of each security group to a file in a text format.

#!/bin/bash
# set variables
instanceid='i-xxxxxxxxxxxxxxxx'
region='us-east-1'
profile='sample'
# log and temp files
output="ec2-sg.log"
tmpfil="ec2-sg.tmp"
# empty log at start
> $output
# get sg ids
aws ec2 describe-instances \
--instance-ids $instanceid \
--region $region \
--profile $profile \
--query 'Reservations[*].Instances[*].SecurityGroups[*].[GroupId]' --output text > $tmpfil
while read -r id; do
  echo '============================================' >> $output
  echo $id >> $output
  echo '============================================' >> $output
  echo '---------------- INGRESS -------------------' >> $output
  aws ec2 describe-security-groups \
  --group-ids $id \
  --profile $profile \
  --region $region \
  --output text \
  --query 'SecurityGroups[].IpPermissions[].[FromPort,ToPort,IpProtocol,IpRanges[].CidrIp[]|[0]]' >> $output
  echo '---------------- EGRESS --------------------' >> $output
  aws ec2 describe-security-groups \
  --group-ids $id \
  --profile $profile \
  --region $region \
  --output text \
  --query 'SecurityGroups[].IpPermissionsEgress[].[FromPort,ToPort,IpProtocol,IpRanges[].CidrIp[]|[0]]' >> $output
done < $tmpfil

Here’s a sample output.

============================================
sg-xxxxxxxxxxxxxxx
============================================
---------------- INGRESS -------------------
5985    5985    tcp     10.0.0.220/32
10005   10005   tcp     10.0.0.164/32
---------------- EGRESS --------------------
80      80      tcp     10.0.0.14/32
40000   65535   udp     10.0.0.0/8
3389    3389    tcp     10.0.0.96/32
9389    9389    tcp     10.0.0.0/8
5985    5986    tcp     10.0.0.96/32

AWS CLI Contains

The AWS CLI has a not so well-known comparison operator called “contains” which can be used to filter or query the output of your results. In this example, we want to show only instances that were not terminated.

Here’s a query containing “?!contains().”

aws ec2 describe-instances \
--query 'Reservations[*].Instances[?!contains(State.Name, `terminated`)].{Instance:InstanceId}'
--output text

Linux Script

The script command records all your keystrokes and output into a file called typescript.

You can start recording by simply typing script.

script
Script started, file is typescript
df -h
ls -l
exit

Typing exit stops the recording. Just cat typescript to see your recording.

cat typescript

AWS EC2 Describe Snapshots

Here’s how to describe snapshots using query with tags.

aws ec2 describe-snapshots \
--owner-ids xxxxxxxxxxx \
--profile default \
--region us-east-2 \
--output text \
--query "Snapshots[*].[SnapshotId,Tags[?Key=='Name'].Value[] | [0]]"

Result.

snap-xxxxxxxxxxxxxxxxx	Server1
snap-xxxxxxxxxxxxxxxxx	Server2

Split A Large File

How to split a large file. 

split -b 500MB httpd.log
ll -lh
total 1.9G
-rw-r--r-- 1 root root 954M Mar 25 12:35 httpd.log
-rw-r--r-- 1 root root 477M Mar 25 12:38 xaa
-rw-r--r-- 1 root root 477M Mar 25 12:38 xab

Split with an output file.

split -b 200M httpd.log split.log
ll -lh
total 1.9G
-rw-r--r-- 1 root root 954M Mar 25 12:35 httpd.log
-rw-r--r-- 1 root root 200M Mar 25 12:52 split.logaa
-rw-r--r-- 1 root root 200M Mar 25 12:52 split.logab
-rw-r--r-- 1 root root 200M Mar 25 12:52 split.logac
-rw-r--r-- 1 root root 200M Mar 25 12:52 split.logad
-rw-r--r-- 1 root root 154M Mar 25 12:52 split.logae

AWS CLI EC2 Describe Tags

Here’s how to get a list of EC2 tags.

aws ec2 describe-tags \
--filters "Name=resource-id,Values=i-xxxxxxxxxxxxx" \
--query 'Tags[][Key,Value]'  \
--profile default \
--region us-east-1 \
--output text