• Skip to main content

Uly.me

cloud engineer

  • Home
  • About
  • Search

directories

Fpsync

September 23, 2019

Fpsync is command line tool for synchronizing directories in parallel using fpart and rsync tools. You can specify a number of concurrent sync jobs, number of files per sync job, and the maximum byte size per sync among other things. Fpsync is believed to be 4 to 5 times faster than rsync. Fpsync makes sense when syncing massive drives with thousands of directories and small files.

To install fpsync.

apt install fpart

apt install fpart

Fpsync with 8 parallel jobs.

log='/root/fpsync.log'
fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log

log='/root/fpsync.log' fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log

A sample Script with timestamps to display elapse time.

#!/bin/bash
log='/root/fpsync.log'
start=$(date)
begin=$(date +%s)
echo 'Start: '$start > $log
fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log
stop=$(date)
end=$(date +%s)
echo 'Stop: '$stop >> $log
elapse=$((end-begin))
 
function show_time () {
    num=$elapse
    min=0
    hour=0
    day=0
    if((num>59));then
        ((sec=num%60))
        ((num=num/60))
        if((num>59));then
            ((min=num%60))
            ((num=num/60))
            if((num>23));then
                ((hour=num%24))
                ((day=num/24))
            else
                ((hour=num))
            fi
        else
            ((min=num))
        fi
    else
        ((sec=num))
    fi
    echo "$day"d "$hour"h "$min"m "$sec"s
}
show_time $elapse >> $log

#!/bin/bash log='/root/fpsync.log' start=$(date) begin=$(date +%s) echo 'Start: '$start > $log fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log stop=$(date) end=$(date +%s) echo 'Stop: '$stop >> $log elapse=$((end-begin)) function show_time () { num=$elapse min=0 hour=0 day=0 if((num>59));then ((sec=num%60)) ((num=num/60)) if((num>59));then ((min=num%60)) ((num=num/60)) if((num>23));then ((hour=num%24)) ((day=num/24)) else ((hour=num)) fi else ((min=num)) fi else ((sec=num)) fi echo "$day"d "$hour"h "$min"m "$sec"s } show_time $elapse >> $log

For comparison, you can substitute fpsync with rysnc and see the performance difference.

fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log
# or
rsync -av /root/tmp1 /root/tmp2/ > /var/null

fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log # or rsync -av /root/tmp1 /root/tmp2/ > /var/null

Filed Under: Cloud, Linux Tagged With: concurrent, directories, fpart, fpsync, parallel, rsync, sync

Diff Between Two Directories

September 2, 2019

Here’s how to get the differences between two directories using diff.

  • -q display only files when they differ
  • -r recursive to all sub-directories
diff -qr /directory-1 /directory-2

diff -qr /directory-1 /directory-2

Filed Under: Linux Tagged With: diff, directories, recursive

List Big Files

March 13, 2019

Find out which files or directories are using up disk resources.

# display the biggest files
find -type f -exec du -Sh {} + | sort -rh | head -n 5
# display the biggest directories
du -Sh | sort -rh | head -5

# display the biggest files find -type f -exec du -Sh {} + | sort -rh | head -n 5 # display the biggest directories du -Sh | sort -rh | head -5

Filed Under: Linux Tagged With: directories, display, files, find, large

  • Go to page 1
  • Go to page 2
  • Go to Next Page »
  • Home
  • About
  • Search

Copyright © 2023