• Skip to main content

Uly.me

cloud engineer

  • Home
  • About
  • Archives

fpart

Fpsync

September 23, 2019

Fpsync is command line tool for synchronizing directories in parallel using fpart and rsync tools. You can specify a number of concurrent sync jobs, number of files per sync job, and the maximum byte size per sync among other things. Fpsync is believed to be 4 to 5 times faster than rsync. Fpsync makes sense when syncing massive drives with thousands of directories and small files.

To install fpsync.

apt install fpart

apt install fpart

Fpsync with 8 parallel jobs.

log='/root/fpsync.log'
fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log

log='/root/fpsync.log' fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log

A sample Script with timestamps to display elapse time.

#!/bin/bash
log='/root/fpsync.log'
start=$(date)
begin=$(date +%s)
echo 'Start: '$start > $log
fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log
stop=$(date)
end=$(date +%s)
echo 'Stop: '$stop >> $log
elapse=$((end-begin))
 
function show_time () {
    num=$elapse
    min=0
    hour=0
    day=0
    if((num>59));then
        ((sec=num%60))
        ((num=num/60))
        if((num>59));then
            ((min=num%60))
            ((num=num/60))
            if((num>23));then
                ((hour=num%24))
                ((day=num/24))
            else
                ((hour=num))
            fi
        else
            ((min=num))
        fi
    else
        ((sec=num))
    fi
    echo "$day"d "$hour"h "$min"m "$sec"s
}
show_time $elapse >> $log

#!/bin/bash log='/root/fpsync.log' start=$(date) begin=$(date +%s) echo 'Start: '$start > $log fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log stop=$(date) end=$(date +%s) echo 'Stop: '$stop >> $log elapse=$((end-begin)) function show_time () { num=$elapse min=0 hour=0 day=0 if((num>59));then ((sec=num%60)) ((num=num/60)) if((num>59));then ((min=num%60)) ((num=num/60)) if((num>23));then ((hour=num%24)) ((day=num/24)) else ((hour=num)) fi else ((min=num)) fi else ((sec=num)) fi echo "$day"d "$hour"h "$min"m "$sec"s } show_time $elapse >> $log

For comparison, you can substitute fpsync with rysnc and see the performance difference.

fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log
# or
rsync -av /root/tmp1 /root/tmp2/ > /var/null

fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log # or rsync -av /root/tmp1 /root/tmp2/ > /var/null

Filed Under: Cloud, Linux Tagged With: concurrent, directories, fpart, fpsync, parallel, rsync, sync

  • Home
  • About
  • Archives

Copyright © 2023