cloud engineer
Here’s how to copy a directory structure from one directory to another without the files.
Copy directory structure of DIR1 to DIR2.
find /DIR1 -type d > /tmp/dirs.txt sed -i 's/DIR1/DIR2/' /tmp/dirs.txt xargs mkdir -p < /tmp/dirs.txt chown -R USER:GROUP /DIR2 |
find /DIR1 -type d > /tmp/dirs.txt sed -i 's/DIR1/DIR2/' /tmp/dirs.txt xargs mkdir -p < /tmp/dirs.txt chown -R USER:GROUP /DIR2
Here’s how to download/upload files using gcloud compute scp.
Make sure you are authenticated.
gcloud auth login |
gcloud auth login
How to download.
gcloud compute scp --recurse your-server:/home/username/yaml.tar.gz . \ --project your-project-id \ --zone us-central1-a \ --internal-ip |
gcloud compute scp --recurse your-server:/home/username/yaml.tar.gz . \ --project your-project-id \ --zone us-central1-a \ --internal-ip
How to upload.
gcloud compute scp --recurse yaml.tar.gz your-server:/home/username/ \ --project your-project-id \ --zone us-central1-a \ --internal-ip |
gcloud compute scp --recurse yaml.tar.gz your-server:/home/username/ \ --project your-project-id \ --zone us-central1-a \ --internal-ip
How to zip multiple files into one zip file.
Using zip.
zip test.zip *.txt |
zip test.zip *.txt
Using tar.
tar cvzf test.tar.gz *.txt |
tar cvzf test.tar.gz *.txt
Untar in current directory or specify another directory.
tar xvzf test.tar.gz . tar xvzf test.tar.gz -C /path/to/dir |
tar xvzf test.tar.gz . tar xvzf test.tar.gz -C /path/to/dir
The /var/log/audit/audit.log files were not being rotated. The files in the directory grew so large that it created disk space issues on /. Here’s the fix. In the /etc/audit/auditd.conf file, change max_log_file_action from “ignore” to “rotate.” Instead of hundreds of files being kept, it will be rotated up to 5 files.
max_log_file_action = rotate
num_logs = 5 |
max_log_file_action = rotate num_logs = 5
Restart service to take effect. The old files will be deleted.
service auditd stop service auditd start service auditd status |
service auditd stop service auditd start service auditd status
Disk space went from 82% down to 25%.
Here’s how to copy multiple files recursively using AWS CLI. The S3 cp command by default only copies a single file. To copy multiple files, you have to use the –recursive option along with –exclude and –include. In this example, we will exclude every file, but include only files with a json extension.
aws s3 cp /tmp/folder s3://bucket/folder \ --recursive --exclude "*" --include "*.json" |
aws s3 cp /tmp/folder s3://bucket/folder \ --recursive --exclude "*" --include "*.json"
The result:
upload: ./abc.json to s3://bucket/folder/abc.json upload: ./xyz.json to s3://bucket/folder/xyz.json |
upload: ./abc.json to s3://bucket/folder/abc.json upload: ./xyz.json to s3://bucket/folder/xyz.json
If you have video files that are formatted in MPEG-2, video files with a .m2ts extension, you can convert them to MP4 using ffmpeg.
ffmpeg -i input.ts -c:v libx264 -c:a aac output.mp4 |
ffmpeg -i input.ts -c:v libx264 -c:a aac output.mp4
The video is encoded using open format H.264, while audio is encoded using AAC.
Find out which files or directories are using up disk resources.
# display the biggest files find -type f -exec du -Sh {} + | sort -rh | head -n 5 # display the biggest directories du -Sh | sort -rh | head -5 |
# display the biggest files find -type f -exec du -Sh {} + | sort -rh | head -n 5 # display the biggest directories du -Sh | sort -rh | head -5