• Skip to main content

Uly.me

cloud engineer

  • Home
  • About
  • Archives

sync

FFMPEG Offset Audio

June 19, 2020

If your audio is out of sync, you can fix using ffmpeg.

ffmpeg -i source.mp4 \
-i livestream.mp4 \
-map 0:0 -map 1:1 \
-acodec copy \
-vcodec copy \
-itsoffset 0.100 \
output.mp4

ffmpeg -i source.mp4 \ -i livestream.mp4 \ -map 0:0 -map 1:1 \ -acodec copy \ -vcodec copy \ -itsoffset 0.100 \ output.mp4

The ffmpeg fix is fast. A 1GB file took less than 30 seconds to run. Adjust -tsoffset until you find the sweet spot.

Filed Under: Linux Tagged With: audio, ffmpeg, offset, sync, video

FFMPEG Fix Audio Sync

June 14, 2020

If you have a video with audio sync issues, you can easily fix it using ffmpeg. It takes 2 inputs: the filename and delay: 0.100 is 100 ms or 0.1 seconds. Output file has -fixed appended to it. It’s fast. Only takes 2-3 seconds to process a 1.5 hour recording.

Here’s the script.

ffmpeg -i livestream.mp4 \
-itsoffset 0.100 \
-i livestream.mp4 \
-map 0:0 -map 1:1 \
-acodec copy \
-vcodec copy \
livestream-fixed.mp4

ffmpeg -i livestream.mp4 \ -itsoffset 0.100 \ -i livestream.mp4 \ -map 0:0 -map 1:1 \ -acodec copy \ -vcodec copy \ livestream-fixed.mp4

Filed Under: Linux Tagged With: audio, delay, ffmpeg, fix, sync

AWS S3 Sync Between Accounts

April 29, 2020

Here’s how to sync S3 buckets between 2 different AWS accounts. Assuming buckets are already created.

  1. Setup bucket permissions in Account A
  2. Setup IAM user with permissions in Account B
  3. Setup bucket permissions in Account B
  4. Run S3 sync from Account B.

Account A bucket permissions. Account and user are from Account B.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::222222222222:user/Jane"},
            "Action": ["s3:ListBucket","s3:GetObject"],
            "Resource": [
                "arn:aws:s3:::awsexamplesourcebucket/*",
                "arn:aws:s3:::awsexamplesourcebucket"
            ]
        }
    ]
}

{ "Version": "2012-10-17", "Statement": [ { "Sid": "DelegateS3Access", "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::222222222222:user/Jane"}, "Action": ["s3:ListBucket","s3:GetObject"], "Resource": [ "arn:aws:s3:::awsexamplesourcebucket/*", "arn:aws:s3:::awsexamplesourcebucket" ] } ] }

Create IAM user (Jane) in Account B

aws iam create-user --user-name Jane

aws iam create-user --user-name Jane

Give IAM user (Jane) access to both buckets.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::awsexamplesourcebucket",
                "arn:aws:s3:::awsexamplesourcebucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::awsexampledestinationbucket",
                "arn:aws:s3:::awsexampledestinationbucket/*"
            ]
        }
    ]
}

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetObject" ], "Resource": [ "arn:aws:s3:::awsexamplesourcebucket", "arn:aws:s3:::awsexamplesourcebucket/*" ] }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:PutObject", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::awsexampledestinationbucket", "arn:aws:s3:::awsexampledestinationbucket/*" ] } ] }

Sync the buckets

aws s3 sync s3://awsexamplesourcebucket s3://awsexampledestinationbucket

aws s3 sync s3://awsexamplesourcebucket s3://awsexampledestinationbucket

Filed Under: Cloud Tagged With: accounts, aws, copy, multiple, s3, sync

Fpsync

September 23, 2019

Fpsync is command line tool for synchronizing directories in parallel using fpart and rsync tools. You can specify a number of concurrent sync jobs, number of files per sync job, and the maximum byte size per sync among other things. Fpsync is believed to be 4 to 5 times faster than rsync. Fpsync makes sense when syncing massive drives with thousands of directories and small files.

To install fpsync.

apt install fpart

apt install fpart

Fpsync with 8 parallel jobs.

log='/root/fpsync.log'
fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log

log='/root/fpsync.log' fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log

A sample Script with timestamps to display elapse time.

#!/bin/bash
log='/root/fpsync.log'
start=$(date)
begin=$(date +%s)
echo 'Start: '$start > $log
fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log
stop=$(date)
end=$(date +%s)
echo 'Stop: '$stop >> $log
elapse=$((end-begin))
 
function show_time () {
    num=$elapse
    min=0
    hour=0
    day=0
    if((num>59));then
        ((sec=num%60))
        ((num=num/60))
        if((num>59));then
            ((min=num%60))
            ((num=num/60))
            if((num>23));then
                ((hour=num%24))
                ((day=num/24))
            else
                ((hour=num))
            fi
        else
            ((min=num))
        fi
    else
        ((sec=num))
    fi
    echo "$day"d "$hour"h "$min"m "$sec"s
}
show_time $elapse >> $log

#!/bin/bash log='/root/fpsync.log' start=$(date) begin=$(date +%s) echo 'Start: '$start > $log fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log stop=$(date) end=$(date +%s) echo 'Stop: '$stop >> $log elapse=$((end-begin)) function show_time () { num=$elapse min=0 hour=0 day=0 if((num>59));then ((sec=num%60)) ((num=num/60)) if((num>59));then ((min=num%60)) ((num=num/60)) if((num>23));then ((hour=num%24)) ((day=num/24)) else ((hour=num)) fi else ((min=num)) fi else ((sec=num)) fi echo "$day"d "$hour"h "$min"m "$sec"s } show_time $elapse >> $log

For comparison, you can substitute fpsync with rysnc and see the performance difference.

fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log
# or
rsync -av /root/tmp1 /root/tmp2/ > /var/null

fpsync -n 8 -v /root/tmp1/ /root/tmp2/ >> $log # or rsync -av /root/tmp1 /root/tmp2/ > /var/null

Filed Under: Cloud, Linux Tagged With: concurrent, directories, fpart, fpsync, parallel, rsync, sync

  • Home
  • About
  • Archives

Copyright © 2023