• Skip to main content

Uly.me

cloud engineer

  • Home
  • About
  • Archives

files

Copy Directory Structure

August 25, 2022

Here’s how to copy a directory structure from one directory to another without the files.

Copy directory structure of DIR1 to DIR2.

find /DIR1 -type d > /tmp/dirs.txt
sed -i 's/DIR1/DIR2/' /tmp/dirs.txt 
xargs mkdir -p < /tmp/dirs.txt
chown -R USER:GROUP /DIR2

find /DIR1 -type d > /tmp/dirs.txt sed -i 's/DIR1/DIR2/' /tmp/dirs.txt xargs mkdir -p < /tmp/dirs.txt chown -R USER:GROUP /DIR2

Filed Under: Linux Tagged With: copy, directory, files, structure, without

GCP gcloud compute scp

January 31, 2022

Here’s how to download/upload files using gcloud compute scp.

Make sure you are authenticated.

gcloud auth login

gcloud auth login

How to download.

gcloud compute scp --recurse your-server:/home/username/yaml.tar.gz . \
--project your-project-id \
--zone us-central1-a \
--internal-ip

gcloud compute scp --recurse your-server:/home/username/yaml.tar.gz . \ --project your-project-id \ --zone us-central1-a \ --internal-ip

How to upload.

gcloud compute scp --recurse yaml.tar.gz your-server:/home/username/ \
--project your-project-id \
--zone us-central1-a \
--internal-ip

gcloud compute scp --recurse yaml.tar.gz your-server:/home/username/ \ --project your-project-id \ --zone us-central1-a \ --internal-ip

Filed Under: Cloud Tagged With: compute, download, files, gcloud, scp, upload

zip multiple files into one zip

December 31, 2021

How to zip multiple files into one zip file.

Using zip.

zip test.zip *.txt

zip test.zip *.txt

Using tar.

tar cvzf test.tar.gz *.txt

tar cvzf test.tar.gz *.txt

Untar in current directory or specify another directory.

tar xvzf test.tar.gz .
tar xvzf test.tar.gz -C /path/to/dir

tar xvzf test.tar.gz . tar xvzf test.tar.gz -C /path/to/dir

Filed Under: Linux Tagged With: directory, files, gzip, multiple, tar, zip

Audit Log Files

November 10, 2020

The /var/log/audit/audit.log files were not being rotated. The files in the directory grew so large that it created disk space issues on /. Here’s the fix. In the /etc/audit/auditd.conf file, change max_log_file_action from “ignore” to “rotate.” Instead of hundreds of files being kept, it will be rotated up to 5 files.

max_log_file_action = rotate
num_logs = 5

max_log_file_action = rotate num_logs = 5

Restart service to take effect. The old files will be deleted.

service auditd stop
service auditd start
service auditd status

service auditd stop service auditd start service auditd status

Disk space went from 82% down to 25%.

Filed Under: Linux Tagged With: audit, files, log, out of disk space, rotate

AWS CLI S3 Recursive Copy

February 2, 2020

Here’s how to copy multiple files recursively using AWS CLI. The S3 cp command by default only copies a single file. To copy multiple files, you have to use the –recursive option along with –exclude and –include. In this example, we will exclude every file, but include only files with a json extension.

aws s3 cp /tmp/folder s3://bucket/folder \
  --recursive
  --exclude "*"
  --include "*.json"

aws s3 cp /tmp/folder s3://bucket/folder \ --recursive --exclude "*" --include "*.json"

The result:

upload: ./abc.json to s3://bucket/folder/abc.json    
upload: ./xyz.json to s3://bucket/folder/xyz.json

upload: ./abc.json to s3://bucket/folder/abc.json upload: ./xyz.json to s3://bucket/folder/xyz.json

Filed Under: Cloud Tagged With: aws, cp, exclude, files, include, multiple, recursive, s3

FFMPEG Convert TS to MP4

December 4, 2019

If you have video files that are formatted in MPEG-2, video files with a .m2ts extension, you can convert them to MP4 using ffmpeg.

ffmpeg -i input.ts -c:v libx264 -c:a aac output.mp4

ffmpeg -i input.ts -c:v libx264 -c:a aac output.mp4

The video is encoded using open format H.264, while audio is encoded using AAC.

Filed Under: Linux, Misc Tagged With: .m2ts, convert, ffmpeg, files, mp4, video

List Big Files

March 13, 2019

Find out which files or directories are using up disk resources.

# display the biggest files
find -type f -exec du -Sh {} + | sort -rh | head -n 5
# display the biggest directories
du -Sh | sort -rh | head -5

# display the biggest files find -type f -exec du -Sh {} + | sort -rh | head -n 5 # display the biggest directories du -Sh | sort -rh | head -5

Filed Under: Linux Tagged With: directories, display, files, find, large

  • Home
  • About
  • Archives

Copyright © 2023