• Skip to main content

Uly.me

cloud engineer

  • Home
  • About
  • Archives

directories

Fpsync

September 23, 2019

Fpsync is command line tool for synchronizing directories in parallel using fpart and rsync tools. You can specify a number of concurrent sync jobs, number of files per sync job, and the maximum byte size per sync among other things. Fpsync is believed to be 4 to 5 times faster than rsync. Fpsync makes sense when syncing massive drives with thousands of directories and small files.

To install fpsync.

apt install fpart

apt install fpart

Fpsync with 8 parallel jobs.

log='/root/fpsync.log'
fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log

log='/root/fpsync.log' fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log

A sample Script with timestamps to display elapse time.

#!/bin/bash
log='/root/fpsync.log'
start=$(date)
begin=$(date +%s)
echo 'Start: '$start > $log
fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log
stop=$(date)
end=$(date +%s)
echo 'Stop: '$stop >> $log
elapse=$((end-begin))
 
function show_time () {
    num=$elapse
    min=0
    hour=0
    day=0
    if((num>59));then
        ((sec=num%60))
        ((num=num/60))
        if((num>59));then
            ((min=num%60))
            ((num=num/60))
            if((num>23));then
                ((hour=num%24))
                ((day=num/24))
            else
                ((hour=num))
            fi
        else
            ((min=num))
        fi
    else
        ((sec=num))
    fi
    echo "$day"d "$hour"h "$min"m "$sec"s
}
show_time $elapse >> $log

#!/bin/bash log='/root/fpsync.log' start=$(date) begin=$(date +%s) echo 'Start: '$start > $log fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log stop=$(date) end=$(date +%s) echo 'Stop: '$stop >> $log elapse=$((end-begin)) function show_time () { num=$elapse min=0 hour=0 day=0 if((num>59));then ((sec=num%60)) ((num=num/60)) if((num>59));then ((min=num%60)) ((num=num/60)) if((num>23));then ((hour=num%24)) ((day=num/24)) else ((hour=num)) fi else ((min=num)) fi else ((sec=num)) fi echo "$day"d "$hour"h "$min"m "$sec"s } show_time $elapse >> $log

For comparison, you can substitute fpsync with rysnc and see the performance difference.

fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log
# or
rsync -av /root/tmp1 /root/tmp2/ > /var/null

fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log # or rsync -av /root/tmp1 /root/tmp2/ > /var/null

Filed Under: Cloud, Linux Tagged With: concurrent, directories, fpart, fpsync, parallel, rsync, sync

Diff Between Two Directories

September 2, 2019

Here’s how to get the differences between two directories using diff.

  • -q display only files when they differ
  • -r recursive to all sub-directories
diff -qr /directory-1 /directory-2

diff -qr /directory-1 /directory-2

Filed Under: Linux Tagged With: diff, directories, recursive

List Big Files

March 13, 2019

Find out which files or directories are using up disk resources.

# display the biggest files
find -type f -exec du -Sh {} + | sort -rh | head -n 5
# display the biggest directories
du -Sh | sort -rh | head -5

# display the biggest files find -type f -exec du -Sh {} + | sort -rh | head -n 5 # display the biggest directories du -Sh | sort -rh | head -5

Filed Under: Linux Tagged With: directories, display, files, find, large

List Directories Using Glob

February 15, 2014

If there’s a need for you to list directories on the server, there’s a PHP function called Glob which will find pathnames to match a certain pattern. To make glob read only directories on the server, we can specify in the glob function with the GLOB_ONLYDIR option which will return only directories and ignore files. The asterisk means it will accept any pattern.

glob('*', GLOB_ONLYDIR);

glob('*', GLOB_ONLYDIR);

In the example below, we will specify any directory by assigning an empty or a blank to the $dir variable. We will then use the foreach loop to display all directories found. If we have WordPress installed, we can ignore the WordPress directories by searching for them in the if statement. We will finally echo and print to the screen each individual directory found by our glob function.

Here’s the entire script with HTML included.

<!DOCTYPE html>
<html lang="en">
<head>
  <title>Glob - list directories</title>
  <style>
    li { list-style-type:none; margin-left:-15px; }
  </style>
</head>
<body>
<h1>List directories using Glob</h1>
<ul>
<?php
// set to current directory 
$dir = '';
// directories only. ignore files, etc.
foreach(glob($dir.'*', GLOB_ONLYDIR) as $folder){ 
  // do not include wordpress directories
  if (($folder != 'wp-admin') && ($folder != 'wp-content') && ($folder != 'wp-includes')) { 
    // list directories and their links
    ?> 
    <li><a href="<?php echo $dir."/".$folder;?>"><?php echo $folder;?></li> 
    <?php 
  }
} 
?>
</ul>
</body>
</html>

<!DOCTYPE html> <html lang="en"> <head> <title>Glob - list directories</title> <style> li { list-style-type:none; margin-left:-15px; } </style> </head> <body> <h1>List directories using Glob</h1> <ul> <?php // set to current directory $dir = ''; // directories only. ignore files, etc. foreach(glob($dir.'*', GLOB_ONLYDIR) as $folder){ // do not include wordpress directories if (($folder != 'wp-admin') && ($folder != 'wp-content') && ($folder != 'wp-includes')) { // list directories and their links ?> <li><a href="<?php echo $dir."/".$folder;?>"><?php echo $folder;?></li> <?php } } ?> </ul> </body> </html>

Filed Under: PHP Tagged With: directories, folders, glob

  • Home
  • About
  • Archives

Copyright © 2023