Tag: efs

AWS EFS List on AWS Backup

Here’s how to get a list of EFS on AWS Backup. Displays a list and looks for a specific tag.

#! /bin/bash
 
tmpfil="temp.txt"
output="efs-list.txt"
 
> $output
> $tmpfil
 
declare -a account=("default" "account1" "account2" "account3")
declare -a region=("us-east-1" "us-east-2" "us-west-1" "us-west-2")
 
for i in "${account[@]}"; do
  echo "===================" >> $output
  echo $i >> $output
  echo "===================" >> $output
  for j in "${region[@]}"; do
    echo $j >> $output
    echo "-------------------" >> $output
    aws efs describe-file-systems \
    --query 'FileSystems[*].[FileSystemId]' \
    --profile $i --region $j --output text >> $tmpfil
    while read -r efs; do
      echo $efs >> $output
      sleep 1s
      aws efs describe-tags --file-system-id $efs \
      --query 'Tags[?Key==`aws-backup`].{Name:Key,Value:Value}' \
      --profile $i --region $j --output text >> $output
      echo "..................." >> $output
    done < $tmpfil
    > $tmpfil
  done
done
rm $tmpfil

Output file is efs-list.txt

Migrate EFS to another VPC

Migrating a EFS from one VPC to another is quite simple. There’s no need to delete it. Just click on “manage file systems access” and remove all mount targets and Save. Once the targets are removed, you can choose another VPC from the dropdown. Once a VPC is selected. Add the subnets and security groups. Save and you are done!

EFS CloudFormation

Here’s a simple EFS CloudFormation template with a one mount target.

{
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "EFS example setup",
	"Parameters": {
		"VPC": {
			"Description": "VPC ID",
			"Type": "AWS::EC2::VPC::Id"
		},
		"Subnet": {			
			"Description": "Subnet ID",
			"Type": "AWS::EC2::Subnet::Id"
		},
		"EC2SecurityGroup": {
			"Description": "Security Group for EC2 instance",
			"Type": "AWS::EC2::SecurityGroup::Id"
		}	
	},
	"Resources": {
		"EFSFileSystem": {
			"Type" : "AWS::EFS::FileSystem",
			"Properties" : {
				"FileSystemTags" : [
					{"Key" : "Name", "Value" : {"Ref": "AWS::StackName"}}
				]
			}
		},
		"EFSMountTarget": {
			"Type": "AWS::EFS::MountTarget",
			"Properties": {
				"FileSystemId": {"Ref": "EFSFileSystem"},
				"SubnetId": { "Ref": "Subnet" },
				"SecurityGroups": [{"Ref": "EFSSecurityGroup"}]        
			}
		},
		"EFSSecurityGroup": {
			"Type": "AWS::EC2::SecurityGroup",
			"Properties": {
				"GroupDescription": "Allowing access to EFS",
				"VpcId": {"Ref": "VPC"},
				"SecurityGroupIngress": [{
					"IpProtocol": "tcp",
					"FromPort": 2049,
					"ToPort": 2049,
					"SourceSecurityGroupId": {"Ref": "EC2SecurityGroup"}
				}]
			}
		}				
	}
}

Calculating the Ideal EFS File Size

AWS EFS runs on burst credits when choosing burst mode, which is about 99% of most implementations. The reason for this is that provisioned throughput is very expensive. It’s a $6000 per month commitment. In burst mode, if the EFS volume is too small with heavy IO usage, the burst credits will nose dive. You will eventually run out of burst credits resulting in very poor IO performance. It will be slower than a floppy disk drive. We all know how old that technology is. In order to sustain your burst credits, you’ll need to add data, because more data means you’ll earn burst credits at a higher clip. Here’s how to calculate the ideal file size for an EFS system.

  1. Identify your throughput needs by looking at your historical usage. From the Amazon CloudWatch console, check the sum statistic of the TotalIOBytes metric with daily aggregation, for the past 14 days. Identify the day with the largest value for TotalIOBytes.
  2. Divide this number by 24 hours, 60 minutes, 60 seconds, and 1024 bytes to get the average KiB/second your application required for that day.
  3. Calculate the file system size (in GiB) required to sustain this average throughput by dividing the average throughput number (in KiB/s) by the baseline throughput number (50 KiB/s/GiB) that EFS provides.

For example, let’s say you have a 500 Gib of TotalIOBytes.

500 x 1024 x 1024 x 1024 / (24 x 60 x 60 x 1024) = 6068.15 Kib/s

6068.15 / 50 Kib/s = 121 Gib

You will need about 121 GB of EFS volume size to maintain your burst credits.