Uly.me

cloud engineer

directories

Fpsync

Fpsync is command line tool for synchronizing directories in parallel using fpart and rsync tools. You can specify a number of concurrent sync jobs, number of files per sync job, and the maximum byte size per sync among other things. Fpsync is believed to be 4 to 5 times faster than rsync. Fpsync makes sense when syncing massive drives with thousands of directories and small files.

To install fpsync.

apt install fpart

Fpsync with 8 parallel jobs.

log='/root/fpsync.log'
fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log

A sample Script with timestamps to display elapse time.

#!/bin/bash
log='/root/fpsync.log'
start=$(date)
begin=$(date +%s)
echo 'Start: '$start > $log
fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log
stop=$(date)
end=$(date +%s)
echo 'Stop: '$stop >> $log
elapse=$((end-begin))
 
function show_time () {
    num=$elapse
    min=0
    hour=0
    day=0
    if((num>59));then
        ((sec=num%60))
        ((num=num/60))
        if((num>59));then
            ((min=num%60))
            ((num=num/60))
            if((num>23));then
                ((hour=num%24))
                ((day=num/24))
            else
                ((hour=num))
            fi
        else
            ((min=num))
        fi
    else
        ((sec=num))
    fi
    echo "$day"d "$hour"h "$min"m "$sec"s
}
show_time $elapse >> $log

For comparison, you can substitute fpsync with rysnc and see the performance difference.

fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log
# or
rsync -av /root/tmp1 /root/tmp2/ > /var/null

Diff Between Two Directories

Here’s how to get the differences between two directories using diff.

  • -q display only files when they differ
  • -r recursive to all sub-directories
diff -qr /directory-1 /directory-2

List Big Files

Find out which files or directories are using up disk resources.

# display the biggest files
find -type f -exec du -Sh {} + | sort -rh | head -n 5
# display the biggest directories
du -Sh | sort -rh | head -5

List Directories Using Glob

If there’s a need for you to list directories on the server, there’s a PHP function called Glob which will find pathnames to match a certain pattern. To make glob read only directories on the server, we can specify in the glob function with the GLOB_ONLYDIR option which will return only directories and ignore files. The asterisk means it will accept any pattern.

glob('*', GLOB_ONLYDIR);

In the example below, we will specify any directory by assigning an empty or a blank to the $dir variable. We will then use the foreach loop to display all directories found. If we have WordPress installed, we can ignore the WordPress directories by searching for them in the if statement. We will finally echo and print to the screen each individual directory found by our glob function.

Here’s the entire script with HTML included.

<!DOCTYPE html>
<html lang="en">
<head>
  <title>Glob - list directories</title>
  <style>
    li { list-style-type:none; margin-left:-15px; }
  </style>
</head>
<body>
<h1>List directories using Glob</h1>
<ul>
<?php
// set to current directory 
$dir = '';
// directories only. ignore files, etc.
foreach(glob($dir.'*', GLOB_ONLYDIR) as $folder){ 
  // do not include wordpress directories
  if (($folder != 'wp-admin') && ($folder != 'wp-content') && ($folder != 'wp-includes')) { 
    // list directories and their links
    ?> 
    <li><a href="<?php echo $dir."/".$folder;?>"><?php echo $folder;?></li> 
    <?php 
  }
} 
?>
</ul>
</body>
</html>